Oct 30 21:34:34.622305 ip-10-0-141-82 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Oct 30 21:34:34.622320 ip-10-0-141-82 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Oct 30 21:34:34.622330 ip-10-0-141-82 systemd[1]: kubelet.service: Failed with result 'resources'.
Oct 30 21:34:34.622692 ip-10-0-141-82 systemd[1]: Failed to start Kubernetes Kubelet.
Oct 30 21:34:45.859332 ip-10-0-141-82 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Oct 30 21:34:45.859352 ip-10-0-141-82 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 835dcc53c7fb43b1b3b3f27d56f3a9b3 --
Oct 30 21:36:44.139921 ip-10-0-141-82 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:36:44.539722 ip-10-0-141-82 kubenswrapper[2578]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:36:44.539722 ip-10-0-141-82 kubenswrapper[2578]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:36:44.539722 ip-10-0-141-82 kubenswrapper[2578]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:36:44.539722 ip-10-0-141-82 kubenswrapper[2578]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:36:44.546250 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546149    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.546250 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546250    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.546250 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546254    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.546250 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546258    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546261    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546266    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546269    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546272    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546274    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546277    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546280    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546283    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546286    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546288    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546291    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546293    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546296    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546299    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546301    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546304    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546307    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546310    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546312    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.546416 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546315    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546317    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546320    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546322    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546325    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546327    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546330    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546332    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546335    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546337    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546340    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546343    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546345    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546348    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546352    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546356    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546358    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546361    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546364    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546367    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.546929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546369    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546372    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546374    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546377    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546380    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546383    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546385    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546388    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546390    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546393    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546396    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546399    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546401    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546404    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546406    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546409    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546411    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546414    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546419    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546423    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.547441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546427    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546430    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546433    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546436    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546439    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546443    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546446    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546449    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546453    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546455    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546458    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546461    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546464    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546466    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546469    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546472    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546474    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546477    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546480    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546482    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.547929 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546485    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546488    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546490    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546493    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546495    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546498    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546501    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546503    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546506    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546509    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546513    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546516    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546518    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546521    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546524    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546526    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546529    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546532    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546535    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.548402 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546538    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546540    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546543    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.546546    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548147    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548155    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548158    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548162    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548165    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548168    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548170    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548173    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548175    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548179    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548181    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548184    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548186    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548189    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548192    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548194    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.548862 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548197    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548199    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548202    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548206    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548208    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548211    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548214    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548216    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548219    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548222    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548225    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548227    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548230    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548232    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548235    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548238    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548240    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548243    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548245    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548248    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.549438 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548251    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548253    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548256    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548258    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548261    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548264    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548266    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548269    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548271    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548274    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548276    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548279    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548282    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548284    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548289    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548295    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548299    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548302    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548304    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.549949 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548307    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548310    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548312    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548315    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548317    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548320    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548322    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548325    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548327    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548331    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548333    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548336    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548338    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548341    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548345    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548347    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548351    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548355    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548358    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.550400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548361    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548364    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548367    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548369    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548372    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548375    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548378    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548380    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548383    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548386    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548388    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548391    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548394    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548396    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548399    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548401    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548406    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548410    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548413    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.550866 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548416    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548420    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548423    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548426    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548429    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548431    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548434    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548437    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548439    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548442    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548444    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548447    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.548449    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549006    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549012    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549016    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549019    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549022    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549025    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549027    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.551310 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549030    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549033    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549036    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549038    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549041    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549043    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549046    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549049    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549051    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549054    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549056    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549059    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549061    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549064    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549066    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549069    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549072    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549077    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549081    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.551813 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549084    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549087    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549090    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549093    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549096    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549099    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549101    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549104    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549107    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549109    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549112    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549115    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549117    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549120    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549122    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549125    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549128    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549130    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549133    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549136    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.552259 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549138    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549141    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549143    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549147    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549150    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549152    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549155    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549157    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549159    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549162    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549165    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549167    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549170    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549182    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549188    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549191    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549193    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549196    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549199    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.552755 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549204    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549206    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549209    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549212    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549214    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549217    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549219    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549222    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549225    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549227    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549230    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549233    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549235    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549238    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549240    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549243    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549246    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549248    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549251    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.553243 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549253    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549256    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549259    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549261    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549263    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549268    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549273    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549276    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549280    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549283    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549285    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549288    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549291    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549295    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549297    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549300    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549302    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549305    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549308    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.553696 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549310    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549313    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.549315    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550224    2578 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550241    2578 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550249    2578 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550254    2578 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550259    2578 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550263    2578 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550268    2578 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550273    2578 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550276    2578 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550279    2578 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550283    2578 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550286    2578 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550289    2578 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550293    2578 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550296    2578 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550299    2578 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550303    2578 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550306    2578 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550312    2578 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550315    2578 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:36:44.554169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550318    2578 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550322    2578 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550325    2578 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550331    2578 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550335    2578 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550338    2578 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550341    2578 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550344    2578 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550348    2578 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550351    2578 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550354    2578 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550357    2578 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550362    2578 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550365    2578 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550368    2578 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550371    2578 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550374    2578 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550377    2578 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550382    2578 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550386    2578 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550389    2578 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550392    2578 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550395    2578 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550400    2578 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:36:44.554729 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550404    2578 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550407    2578 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550410    2578 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550413    2578 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550417    2578 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550420    2578 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550423    2578 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550427    2578 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550429    2578 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550433    2578 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550438    2578 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550442    2578 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550445    2578 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550449    2578 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550452    2578 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550456    2578 flags.go:64] FLAG: --help="false"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550459    2578 flags.go:64] FLAG: --hostname-override="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550462    2578 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550465    2578 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550468    2578 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550472    2578 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550475    2578 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550478    2578 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:36:44.555312 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550481    2578 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550484    2578 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550487    2578 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550490    2578 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550493    2578 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550496    2578 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550499    2578 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550502    2578 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550505    2578 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550508    2578 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550511    2578 flags.go:64] FLAG: --lock-file=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550514    2578 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550517    2578 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550520    2578 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550526    2578 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550530    2578 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550533    2578 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550537    2578 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550540    2578 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550544    2578 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550547    2578 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550550    2578 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550555    2578 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550558    2578 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550562    2578 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:36:44.555883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550565    2578 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550568    2578 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550571    2578 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550575    2578 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550578    2578 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550581    2578 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550594    2578 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550603    2578 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550606    2578 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550610    2578 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550613    2578 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550615    2578 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550619    2578 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550622    2578 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550625    2578 flags.go:64] FLAG: --port="10250"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550628    2578 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550631    2578 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d7878499bfd6b86e"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550635    2578 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550638    2578 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550641    2578 flags.go:64] FLAG: --register-node="true"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550644    2578 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550649    2578 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550652    2578 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550656    2578 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550659    2578 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550663    2578 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:36:44.556484 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550666    2578 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550670    2578 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550672    2578 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550676    2578 flags.go:64] FLAG: --runonce="false"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550678    2578 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550681    2578 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550685    2578 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550688    2578 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550691    2578 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550694    2578 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550697    2578 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550701    2578 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550704    2578 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550707    2578 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550710    2578 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550713    2578 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550716    2578 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550719    2578 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550721    2578 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550729    2578 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550732    2578 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550735    2578 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550739    2578 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550742    2578 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550745    2578 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:36:44.557146 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550749    2578 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550752    2578 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550755    2578 flags.go:64] FLAG: --v="2"
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550764    2578 flags.go:64] FLAG: --version="false"
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550769    2578 flags.go:64] FLAG: --vmodule=""
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550774    2578 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.550777    2578 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550924    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550927    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550930    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550933    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550936    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550938    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550941    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550943    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550946    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550948    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550951    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550953    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550956    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550958    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.558236 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550961    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550963    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550966    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550968    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550971    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550976    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550979    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550981    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550984    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550987    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550989    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550995    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.550998    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551000    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551003    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551005    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551008    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551011    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551013    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.559142 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551017    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551021    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551024    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551027    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551030    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551032    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551035    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551038    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551040    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551043    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551047    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551051    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551054    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551057    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551059    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551062    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551065    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551067    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551071    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.559695 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551074    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551076    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551079    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551082    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551084    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551088    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551091    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551093    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551096    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551099    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551101    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551104    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551107    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551110    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551113    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551116    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551118    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551121    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551123    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551126    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.560155 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551128    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551131    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551133    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551136    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551138    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551141    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551144    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551146    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551149    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551152    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551155    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551159    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551161    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551164    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551166    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551169    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551171    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551176    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551179    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.560671 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551182    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551184    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551187    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551190    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551192    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551195    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551198    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551201    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551203    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551206    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551209    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551211    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551214    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551216    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551219    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.551248    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:36:44.561132 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551287    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551293    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551297    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551301    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551303    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551306    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551309    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551312    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551317    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551319    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551322    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551326    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551330    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551333    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551336    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551340    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551343    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551346    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551348    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.561521 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551351    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551354    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551357    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551360    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551363    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551365    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551368    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551371    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551373    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551376    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551378    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551381    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551384    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551386    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551389    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551391    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551394    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551397    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551399    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551402    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.562015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551404    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551408    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551410    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551413    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551415    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551418    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551421    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551423    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551429    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551431    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551434    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551437    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551439    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551442    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551445    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551447    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551449    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551453    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551455    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.562495 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551458    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551461    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551463    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551466    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551469    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551471    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551474    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551476    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551480    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551483    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551486    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551488    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551491    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551493    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551497    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551500    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551503    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551505    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551508    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551511    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.562967 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551513    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551517    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551520    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551522    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551525    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551527    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551530    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551533    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551535    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551538    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551541    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551544    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551546    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551549    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551551    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551554    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551557    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551560    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551562    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551566    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.563450 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551568    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551571    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551573    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551576    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551578    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551581    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551596    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.551599    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.560360    2578 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.560379    2578 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.560398    2578 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.560404    2578 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560467    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560473    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560477    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560480    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560483    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560487    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560490    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.563948 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560493    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560496    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560499    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560502    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560505    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560508    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560512    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560515    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560517    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560520    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560523    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560526    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560529    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560531    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560534    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560537    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560540    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560542    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560545    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560548    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.564441 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560551    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560559    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560562    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560565    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560568    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560571    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560574    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560576    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560579    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560582    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560600    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560603    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560606    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560609    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560611    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560614    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560617    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560619    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560622    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.565072 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560625    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560629    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560633    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560638    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560642    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560645    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560648    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560651    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560653    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560656    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560659    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560662    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560665    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560667    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560670    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560674    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560676    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560679    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560682    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.565517 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560685    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560688    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560690    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560693    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560695    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560698    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560700    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560703    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560705    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560708    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560710    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560713    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560715    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560718    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560722    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560725    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560728    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560730    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560733    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560754    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.566019 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560758    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560761    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560764    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560768    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560772    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560775    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560778    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560781    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560784    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560788    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560790    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560793    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560796    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560798    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560801    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560804    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560806    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560809    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560812    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560814    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.566648 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560817    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.560850    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.560994    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561001    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561004    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561007    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561010    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561014    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561017    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561020    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561022    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561025    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561028    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561031    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561033    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561036    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.567131 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561039    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561042    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561044    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561047    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561050    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561053    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561056    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561058    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561060    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561063    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561066    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561068    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561071    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561073    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561076    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561078    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561081    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561083    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561086    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561088    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.567703 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561091    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561093    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561096    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561099    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561102    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561105    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561108    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561110    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561113    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561116    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561119    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561122    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561126    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561131    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561134    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561138    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561141    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561144    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561147    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.568315 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561149    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561152    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561154    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561157    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561159    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561162    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561164    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561167    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561178    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561183    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561186    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561189    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561192    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561195    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561197    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561200    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561203    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561206    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561209    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.569100 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561212    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561215    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561217    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561220    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561223    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561225    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561228    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561231    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561233    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561236    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561239    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561241    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561244    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561247    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561249    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561252    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561254    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561257    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561260    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561262    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.569715 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561265    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561267    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561270    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561272    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561275    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561277    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561280    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561282    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561285    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561287    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561290    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561293    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561296    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561299    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.561323    2578 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561363    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:36:44.570263 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561367    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561370    2578 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561373    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561376    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561378    2578 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561381    2578 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561384    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561386    2578 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561389    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561392    2578 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561395    2578 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561397    2578 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561400    2578 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561402    2578 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561405    2578 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561408    2578 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561410    2578 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561413    2578 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561415    2578 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561418    2578 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:36:44.570675 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561420    2578 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561423    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561425    2578 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561428    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561430    2578 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561433    2578 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561436    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561438    2578 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561441    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561444    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561447    2578 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561450    2578 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561452    2578 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561455    2578 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561458    2578 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561461    2578 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561463    2578 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561466    2578 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561468    2578 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561471    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:36:44.571170 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561474    2578 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561476    2578 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561479    2578 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561481    2578 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561484    2578 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561486    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561489    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561492    2578 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561494    2578 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561499    2578 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561502    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561504    2578 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561507    2578 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561509    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561512    2578 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561514    2578 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561517    2578 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561520    2578 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561524    2578 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561527    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:36:44.571968 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561529    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561532    2578 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561536    2578 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561539    2578 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561542    2578 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561544    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561547    2578 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561550    2578 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561552    2578 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561555    2578 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561557    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561560    2578 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561562    2578 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561565    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561568    2578 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561571    2578 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561574    2578 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561576    2578 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561580    2578 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:36:44.572491 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561597    2578 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561601    2578 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561606    2578 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561609    2578 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561612    2578 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561614    2578 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561617    2578 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561619    2578 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561622    2578 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561625    2578 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561627    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561631    2578 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561634    2578 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561637    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561641    2578 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561644    2578 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561648    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561651    2578 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561655    2578 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:36:44.572956 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561657    2578 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561660    2578 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561662    2578 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561665    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561667    2578 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561670    2578 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:44.561674    2578 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.561737    2578 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.564699    2578 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.566251    2578 server.go:1014] "Starting client certificate rotation"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.566362    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.566417    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.568230    2578 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:36:44.573400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.569486    2578 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:36:44.579175 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.579154    2578 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:36:44.583019 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.583003    2578 log.go:25] "Validated CRI v1 image API"
Oct 30 21:36:44.583916 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.583884    2578 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:36:44.589022 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.588990    2578 fs.go:135] Filesystem UUIDs: map[166dc5c4-397d-4d6a-844f-e3013f9894e7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f61f6cc5-e4e7-4428-a1ea-fff8a5a72e0c:/dev/nvme0n1p3]
Oct 30 21:36:44.589123 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.589020    2578 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Oct 30 21:36:44.595082 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.594931    2578 manager.go:217] Machine: {Timestamp:2026-10-30 21:36:44.593510554 +0000 UTC m=+0.336528425 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099112 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f9a72f1e34050867a8cf0fd7ebb7 SystemUUID:ec22f9a7-2f1e-3405-0867-a8cf0fd7ebb7 BootID:835dcc53-c7fb-43b1-b3b3-f27d56f3a9b3 Filesystems:[{Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:06:ff:cc:e2:39:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:06:ff:cc:e2:39:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:32:58:4d:46:c9:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:36:44.595769 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.595754    2578 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:36:44.595915 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.595895    2578 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:36:44.600138 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.599939    2578 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:36:44.600294 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.600141    2578 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-82.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:36:44.600347 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.600297    2578 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:36:44.600347 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.600307    2578 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:36:44.600418 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.600347    2578 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:36:44.600418 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.600365    2578 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:36:44.601535 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.601523    2578 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:36:44.605712 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.605697    2578 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:36:44.605840 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.605831    2578 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:36:44.606671 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.606661    2578 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:36:44.606715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.606676    2578 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:36:44.606715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.606686    2578 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:36:44.606715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.606695    2578 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:36:44.606715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.606705    2578 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:36:44.607097 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.607074    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:36:44.609507 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.609457    2578 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:36:44.616572 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.616550    2578 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:36:44.617653 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617638    2578 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:36:44.617725 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617671    2578 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:36:44.617846 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617833    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:36:44.617880 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617850    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:36:44.617880 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617857    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:36:44.617880 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617864    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:36:44.617880 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617869    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:36:44.617880 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.617875    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:36:44.618405 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618396    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:36:44.618433 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618405    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:36:44.618433 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618413    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:36:44.618433 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618421    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:36:44.618433 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618430    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:36:44.618603 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.618438    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:36:44.619081 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.619071    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:36:44.619081 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.619080    2578 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:36:44.623093 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.623069    2578 server.go:1263] "Started kubelet"
Oct 30 21:36:44.623240 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.623170    2578 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:36:44.623345 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.623208    2578 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:36:44.623345 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.623264    2578 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:36:44.623965 ip-10-0-141-82 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:36:44.624811 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.624795    2578 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:36:44.626684 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.626669    2578 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:36:44.631544 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.631507    2578 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Oct 30 21:36:44.632928 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.632911    2578 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:36:44.634010 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.633991    2578 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:36:44.634188 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.634131    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:44.634786 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.634763    2578 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:36:44.634786 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.634766    2578 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:36:44.634923 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.634799    2578 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:36:44.634923 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.634903    2578 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:36:44.634923 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.634912    2578 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:36:44.635129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635109    2578 factory.go:55] Registering systemd factory
Oct 30 21:36:44.635129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635131    2578 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:36:44.635360 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635346    2578 factory.go:153] Registering CRI-O factory
Oct 30 21:36:44.635360 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635361    2578 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:36:44.635442 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635403    2578 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:36:44.635442 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635418    2578 factory.go:103] Registering Raw factory
Oct 30 21:36:44.635442 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.635427    2578 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:36:44.636525 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.636511    2578 manager.go:319] Starting recovery of all containers
Oct 30 21:36:44.650286 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.650262    2578 manager.go:324] Recovery completed
Oct 30 21:36:44.655527 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.655509    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.657555 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.657524    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.657653 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.657574    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.657653 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.657617    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.658259 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.658243    2578 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:36:44.658259 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.658257    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.658422 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.658256    2578 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:36:44.658422 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.658339    2578 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:36:44.659038 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.659012    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.659092 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.659047    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.659092 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.659061    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.660573 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.660558    2578 policy_none.go:50] "Start"
Oct 30 21:36:44.660664 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.660579    2578 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:36:44.660664 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.660609    2578 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:36:44.662679 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.662665    2578 policy_none.go:44] "Start"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.706968    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.707311    2578 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.708333    2578 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.708351    2578 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.708371    2578 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.708410    2578 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.710777    2578 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.710808    2578 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.710819    2578 server.go:87] "Starting device plugin registration server"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.711103    2578 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.711113    2578 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.711420    2578 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.711499    2578 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.711506    2578 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.712055    2578 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:36:44.714407 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.712133    2578 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:44.760945 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.760909    2578 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-82.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Oct 30 21:36:44.806620 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.806365    2578 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-82.us-west-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Oct 30 21:36:44.807302 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.806348    2578 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-82.us-west-2.compute.internal.18e36c58a616f2bb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-82.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-82.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:36:44.623033019 +0000 UTC m=+0.366050889,LastTimestamp:2026-10-30 21:36:44.623033019 +0000 UTC m=+0.366050889,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-82.us-west-2.compute.internal,}"
Oct 30 21:36:44.808509 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.808490    2578 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"]
Oct 30 21:36:44.808556 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.808548    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.809581 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.809564    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.809701 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.809613    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.809701 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.809628    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.811842 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.811824    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.811942 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.811893    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.812041 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812028    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.812105 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812060    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.812689 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812661    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.812689 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812674    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.812689 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812690    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812699    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812705    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812713    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812730    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812740    2578 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812749    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.812876 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.812797    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.815218 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.815199    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.815305 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.815238    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:44.816007 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.815992    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:44.816091 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.816018    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:44.816091 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.816029    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:44.829398 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.829376    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-82.us-west-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Oct 30 21:36:44.829672 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.829649    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Oct 30 21:36:44.829774 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.829724    2578 reflector.go:204] "Failed to watch" err="failed to list *v1.RuntimeClass: runtimeclasses.node.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"runtimeclasses\" in API group \"node.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass"
Oct 30 21:36:44.836041 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.836024    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.836124 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.836057    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.836124 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.836084    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.844366 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.844348    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.847138 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.847116    2578 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.855552 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.855535    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:36:44.855630 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.855576    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-cfjlj"
Oct 30 21:36:44.914937 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.914907    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:36:44.923800 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.923777    2578 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.923863 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:44.923804    2578 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-82.us-west-2.compute.internal\": node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:44.936679 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936650    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.936781 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936684    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.936781 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.936781 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936765    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.936781 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936770    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:44.936926 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:44.936773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:45.029647 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.029615    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.130260 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.130198    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.148340 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.148315    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:45.153400 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.153378    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:45.231291 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.231255    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.332627 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.332577    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.433328 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.433234    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.533478 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.533445    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.567407 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.567378    2578 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Oct 30 21:36:45.567883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.567616    2578 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Oct 30 21:36:45.567883 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.567572    2578 event.go:368] "Unable to write event (may retry after sleeping)" err="Patch \"https://ac60eab6f532d4004a67394301a17821-2cf05b1ccc71f3c2.elb.us-east-2.amazonaws.com:6443/api/v1/namespaces/default/events/ip-10-0-141-82.us-west-2.compute.internal.18e36c58a825b768\": read tcp 10.0.141.82:41754->3.151.38.176:6443: use of closed network connection" event="&Event{ObjectMeta:{ip-10-0-141-82.us-west-2.compute.internal.18e36c58a825b768  default   4803 0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-82.us-west-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-82.us-west-2.compute.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-82.us-west-2.compute.internal,},FirstTimestamp:2026-10-30 21:36:44 +0000 UTC,LastTimestamp:2026-10-30 21:36:44.812688014 +0000 UTC m=+0.555705894,Count:5,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-82.us-west-2.compute.internal,}"
Oct 30 21:36:45.633664 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.633624    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.735244 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.735154    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.835695 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.835655    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.862950 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.862917    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:31:45 +0000 UTC" deadline="2028-06-13 07:07:33.968646978 +0000 UTC"
Oct 30 21:36:45.862950 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.862946    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14193h30m48.105703996s"
Oct 30 21:36:45.884865 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.884833    2578 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:36:45.886080 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.886058    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:36:45.886210 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.886096    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:36:45.886210 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.886106    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:36:45.936859 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:45.936827    2578 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:36:45.939175 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:45.939160    2578 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:36:46.034497 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.034400    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:46.078212 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.078181    2578 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:36:46.104920 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.104896    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:36:46.106676 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.106652    2578 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:36:46.170715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.170675    2578 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:36:46.356101 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.356075    2578 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:36:46.419884 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:46.419849    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod845d50ae5087e4cd62e2c4035d1ecf3f.slice/crio-1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d WatchSource:0}: Error finding container 1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d: Status 404 returned error can't find the container with id 1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d
Oct 30 21:36:46.420459 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:46.420436    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b11aa0892d28c2064080199cdadd8ae.slice/crio-c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c WatchSource:0}: Error finding container c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c: Status 404 returned error can't find the container with id c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c
Oct 30 21:36:46.426671 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.426648    2578 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:36:46.609269 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.609180    2578 apiserver.go:52] "Watching apiserver"
Oct 30 21:36:46.634131 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.634089    2578 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Oct 30 21:36:46.713878 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.713817    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerStarted","Data":"c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c"}
Oct 30 21:36:46.714928 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.714903    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"845d50ae5087e4cd62e2c4035d1ecf3f","Type":"ContainerStarted","Data":"1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d"}
Oct 30 21:36:46.722921 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.722895    2578 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:36:46.723077 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.723063    2578 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:36:46.723359 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.723344    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-6pwrc","openshift-image-registry/node-ca-7rdhn","openshift-multus/multus-additional-cni-plugins-k4kwl","openshift-multus/multus-z6crj","openshift-multus/network-metrics-daemon-x465r","openshift-dns/node-resolver-4cmqj","openshift-network-operator/iptables-alerter-pz6fg","openshift-ovn-kubernetes/ovnkube-node-99kdf","kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal","openshift-network-diagnostics/network-check-target-fdqv5","kube-system/konnectivity-agent-sxl4l","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b","openshift-cluster-node-tuning-operator/tuned-fgd96"]
Oct 30 21:36:46.727320 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.727300    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.729577 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.729550    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.732041 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.732022    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.734304 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.734284    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.736678 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.736645    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.736780 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.736724    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:46.738904 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.738884    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.741259 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.741242    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.744658 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.744634    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.744789 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.744761    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:46.745515 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745487    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.745598 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745530    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.745598 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745559    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.745598 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745586    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djmm6\" (UniqueName: \"kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745613    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745637    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745687    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745734    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.745766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745763    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745837    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745868    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jw5z\" (UniqueName: \"kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745885    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745901    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745918    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67xtz\" (UniqueName: \"kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745937    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw5h\" (UniqueName: \"kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.745986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.745964    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746005    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746051    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746076    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746095    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746121    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746145    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.746206 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746171    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746213    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746263    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.746413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746294    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.746413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746328    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746346    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.746626 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746351    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.746626 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746513    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746626 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746556    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746626 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746584    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746783 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746783 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746647    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.746783 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.746665    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tq8jv\" (UniqueName: \"kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.749089 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.749071    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.751519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.751501    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:46.751627 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.751573    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:46.754028 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.754010    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.785494 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.785330    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.785494 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.785412    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:36:46.785494 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.785430    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.785494 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.785475    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5bph\""
Oct 30 21:36:46.786556 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.786537    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:36:46.786969 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.786947    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.787085 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.786982    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.787187 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.787173    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfsxs\""
Oct 30 21:36:46.789537 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.789523    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.789829 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.789816    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:36:46.790063 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.790050    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:36:46.790101 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.790071    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.790134 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.790105    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hkmjr\""
Oct 30 21:36:46.790134 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.790127    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:36:46.791480 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.791467    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:36:46.791611 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.791598    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hcvt7\""
Oct 30 21:36:46.796107 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.796091    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.796281 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.796269    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t84mq\""
Oct 30 21:36:46.796371 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.796358    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.803510 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.803492    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlst6\""
Oct 30 21:36:46.803651 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.803573    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5bth\""
Oct 30 21:36:46.804078 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.804061    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:36:46.804078 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.804070    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.804178 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.804101    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.804354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.804338    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.804390 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.804351    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.806272 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.806260    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:36:46.836519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.836494    2578 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:36:46.839762 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839745    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:36:46.839854 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839764    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:36:46.839854 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839770    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:36:46.839854 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839810    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:36:46.839854 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839830    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:36:46.839854 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.839833    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lt5wb\""
Oct 30 21:36:46.840306 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.840290    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:36:46.840390 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.840376    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b6pf6\""
Oct 30 21:36:46.841065 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.841039    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:36:46.841169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.841138    2578 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Oct 30 21:36:46.841230 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.841219    2578 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-hrwmw"
Oct 30 21:36:46.847172 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847151    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.847278 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847177    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:46.847278 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847200    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.847278 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847217    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.847278 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847242    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847278 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847263    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jw5z\" (UniqueName: \"kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847300    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847314    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847262    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847325    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847376    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847376    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847409    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847444    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847477    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.847531 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847516    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847571    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847604    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847662    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847688    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847714    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847721    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847739    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847756    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847763    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847782    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9k26r\" (UniqueName: \"kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.847812    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847814    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.847883    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:47.347862178 +0000 UTC m=+3.090880056 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847898    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847917    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.847994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847976    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.847994    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848012    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65tq4\" (UniqueName: \"kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848033    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848100    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848140    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848242    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848271    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848318    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848348    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848373    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848419    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.848571 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848451    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848422    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848480    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848499    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848511    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848535    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5jns\" (UniqueName: \"kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848552    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848567    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btqkw\" (UniqueName: \"kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848616    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848652    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848655    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848689    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848712    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848824    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848848    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848874    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xtz\" (UniqueName: \"kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848899    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw5h\" (UniqueName: \"kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848926    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848935    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848978    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.848993    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849007    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849019    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849029    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849091    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849122    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849166    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849181    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849211    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849218    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849237    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.849849 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849279    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849301    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849337    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849380    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849406    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849402    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849451    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849469    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849495    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849538    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849575    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849587    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849544    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849627    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849629    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849655    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849680    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849727    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849745    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849778    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849818    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8jv\" (UniqueName: \"kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849868    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849894    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmm6\" (UniqueName: \"kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849918    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849954    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849963    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849965    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.850873 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.849968    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.912103 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.912016    2578 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:36:46.915945 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.915913    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xtz\" (UniqueName: \"kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:46.916129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.915945    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmm6\" (UniqueName: \"kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:46.916129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.915952    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jw5z\" (UniqueName: \"kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:36:46.916129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.915980    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8jv\" (UniqueName: \"kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:46.916129 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.915908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw5h\" (UniqueName: \"kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:46.951037 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951037 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951038    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951056    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951134    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951133    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951158    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951158    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951235 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951215    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951152    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951287    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951341    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951354    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951425    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951448    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951466    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.951519 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951503    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951524    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951557    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951583    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951624    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951649    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951702    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951784    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951811    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951916    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951946    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951969    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.951978    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952005    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952049 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952053    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952076    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952117    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952149    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952171    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952200    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952227    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952236    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952270    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952275    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952278    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952310    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k26r\" (UniqueName: \"kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952284    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952348    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952343    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952430    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952519    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.952537    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:46.952775 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952550    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952579    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952617    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:46.952659    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:47.452640932 +0000 UTC m=+3.195658810 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952676    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952708    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952679    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952744    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tq4\" (UniqueName: \"kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952767    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952794    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952818    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952851    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952820    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952910    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952916    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952795    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952952    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.954807 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952980    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953001    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953031    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.952957    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953057    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953083    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jns\" (UniqueName: \"kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953092    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.953113    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954046    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954116    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqkw\" (UniqueName: \"kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954150    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954441    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954780    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954787    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.954773    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.955034    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.955061    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:46.955624 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.955382    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:46.956487 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.956396    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:46.956677 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:46.956637    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:47.016329 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.016281    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:36:47.016329 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.016317    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:36:47.016329 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.016336    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:47.016577 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.016420    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:47.516396933 +0000 UTC m=+3.259414814 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:47.018958 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.018924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tq4\" (UniqueName: \"kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:47.019103 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.018924    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k26r\" (UniqueName: \"kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:47.019581 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.019557    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqkw\" (UniqueName: \"kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:47.020352 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.020329    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jns\" (UniqueName: \"kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:47.040562 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.040534    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:36:47.051222 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.051196    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ee90e5a_7c9f_4b24_841c_86b50175efd6.slice/crio-6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f WatchSource:0}: Error finding container 6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f: Status 404 returned error can't find the container with id 6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f
Oct 30 21:36:47.052638 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.052586    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:36:47.061506 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.061484    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79565b90_0624_4317_983b_6f44b73c6f78.slice/crio-b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba WatchSource:0}: Error finding container b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba: Status 404 returned error can't find the container with id b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba
Oct 30 21:36:47.067928 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.067907    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z6crj"
Oct 30 21:36:47.068988 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.068967    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:36:47.072734 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.072714    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:36:47.075109 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.075086    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:36:47.078464 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.078446    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:36:47.080691 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.080652    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e474a6c_d260_4e3e_bdb9_d3a00d5e3749.slice/crio-45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3 WatchSource:0}: Error finding container 45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3: Status 404 returned error can't find the container with id 45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3
Oct 30 21:36:47.084972 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.084892    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d6a00c1_5c3d_48e9_974d_9ac8477c2f79.slice/crio-3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc WatchSource:0}: Error finding container 3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc: Status 404 returned error can't find the container with id 3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc
Oct 30 21:36:47.085940 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.085866    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f98edf_bb0a_422e_ab8f_c1f935774f2a.slice/crio-4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328 WatchSource:0}: Error finding container 4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328: Status 404 returned error can't find the container with id 4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328
Oct 30 21:36:47.090835 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.090809    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode021af5f_e7c2_474e_aff2_7cc532fbcaf0.slice/crio-127ce611219e704e1b423deff57b1efa74ce03be7c4dcbb8828943f58734e662 WatchSource:0}: Error finding container 127ce611219e704e1b423deff57b1efa74ce03be7c4dcbb8828943f58734e662: Status 404 returned error can't find the container with id 127ce611219e704e1b423deff57b1efa74ce03be7c4dcbb8828943f58734e662
Oct 30 21:36:47.091787 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.091767    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0d8c917_e87e_4304_bec9_66f213536b88.slice/crio-0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018 WatchSource:0}: Error finding container 0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018: Status 404 returned error can't find the container with id 0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018
Oct 30 21:36:47.095732 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.095712    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:47.102866 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.102847    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:36:47.105566 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.105538    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bf2774c_7d35_4bff_98ae_489cc598b066.slice/crio-71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b WatchSource:0}: Error finding container 71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b: Status 404 returned error can't find the container with id 71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b
Oct 30 21:36:47.111583 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:36:47.111558    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71ef1ace_7a0c_4e16_9b99_2cc43e7ebaaa.slice/crio-61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef WatchSource:0}: Error finding container 61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef: Status 404 returned error can't find the container with id 61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef
Oct 30 21:36:47.357813 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.357716    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:47.357965 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.357884    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:47.357965 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.357955    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:48.357939724 +0000 UTC m=+4.100957584 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:47.459396 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.459349    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:47.459556 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.459525    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:47.459649 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.459593    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:48.459578092 +0000 UTC m=+4.202595956 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:47.560797 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.560760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:47.560982 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.560898    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:36:47.560982 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.560913    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:36:47.560982 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.560923    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:47.560982 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:47.560974    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:48.560961191 +0000 UTC m=+4.303979051 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:47.717572 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.717484    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pz6fg" event={"ID":"8ee90e5a-7c9f-4b24-841c-86b50175efd6","Type":"ContainerStarted","Data":"6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f"}
Oct 30 21:36:47.718744 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.718709    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxl4l" event={"ID":"2bf2774c-7d35-4bff-98ae-489cc598b066","Type":"ContainerStarted","Data":"71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b"}
Oct 30 21:36:47.719777 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.719753    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rdhn" event={"ID":"f0d8c917-e87e-4304-bec9-66f213536b88","Type":"ContainerStarted","Data":"0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018"}
Oct 30 21:36:47.720900 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.720878    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgd96" event={"ID":"e021af5f-e7c2-474e-aff2-7cc532fbcaf0","Type":"ContainerStarted","Data":"127ce611219e704e1b423deff57b1efa74ce03be7c4dcbb8828943f58734e662"}
Oct 30 21:36:47.721953 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.721933    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b" event={"ID":"79565b90-0624-4317-983b-6f44b73c6f78","Type":"ContainerStarted","Data":"b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba"}
Oct 30 21:36:47.723124 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.723100    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef"}
Oct 30 21:36:47.724254 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.724233    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerStarted","Data":"4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328"}
Oct 30 21:36:47.725389 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.725368    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4cmqj" event={"ID":"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79","Type":"ContainerStarted","Data":"3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc"}
Oct 30 21:36:47.726510 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.726486    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6crj" event={"ID":"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749","Type":"ContainerStarted","Data":"45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3"}
Oct 30 21:36:47.844018 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.843972    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:31:46 +0000 UTC" deadline="2028-04-07 21:48:56.155282886 +0000 UTC"
Oct 30 21:36:47.844018 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:47.844015    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12600h12m8.31127268s"
Oct 30 21:36:48.366743 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.366697    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:48.366990 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.366866    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:48.366990 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.366938    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:50.366916734 +0000 UTC m=+6.109934615 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:48.467893 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.467842    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:48.468070 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.468001    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:48.468070 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.468063    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:50.468050042 +0000 UTC m=+6.211067916 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:48.568879 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.568836    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:48.569205 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.569055    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:36:48.569205 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.569083    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:36:48.569205 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.569098    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:48.569205 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.569160    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:50.569141249 +0000 UTC m=+6.312159128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:48.711437 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.711359    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:48.711651 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.711505    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:48.711651 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.711597    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:48.711797 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.711697    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:48.711797 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.711764    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:48.711909 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:48.711832    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:48.844700 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.844611    2578 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:31:46 +0000 UTC" deadline="2028-04-14 19:09:50.415776012 +0000 UTC"
Oct 30 21:36:48.844700 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:48.844690    2578 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12765h33m1.571089666s"
Oct 30 21:36:50.384288 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.383708    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:50.384288 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.383879    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:50.384288 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.383943    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:54.383923684 +0000 UTC m=+10.126941556 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:50.485381 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.484760    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:50.485381 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.484940    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:50.485381 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.484996    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:54.484983363 +0000 UTC m=+10.228001222 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:50.585457 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.585416    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:50.585708 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.585689    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:36:50.585773 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.585716    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:36:50.585773 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.585730    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:50.585886 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.585789    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:36:54.585771998 +0000 UTC m=+10.328789861 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:50.711672 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.711600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:50.711826 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.711600    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:50.711826 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.711735    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:50.711826 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.711772    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:50.713233 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:50.713076    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:50.713233 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:50.713197    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:51.765498 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:51.765066    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"845d50ae5087e4cd62e2c4035d1ecf3f","Type":"ContainerStarted","Data":"f2d33c453524c0fe5cb5723ca7077c32bf56ed0719c0084e45db99c8aa89f141"}
Oct 30 21:36:51.771534 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:51.771496    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerStarted","Data":"158a926b33cc7e282cfa6b749be03fd5902196af85b89c408d5906d631bbe6a5"}
Oct 30 21:36:51.785247 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:51.785207    2578 generic.go:356] "Generic (PLEG): container finished" podID="6b11aa0892d28c2064080199cdadd8ae" containerID="d7af3f5fc7a5c7e23cca5703413244b70b22aa32f6cdbeeaea0f9ddfafc2ac1d" exitCode=0
Oct 30 21:36:51.785405 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:51.785266    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerDied","Data":"d7af3f5fc7a5c7e23cca5703413244b70b22aa32f6cdbeeaea0f9ddfafc2ac1d"}
Oct 30 21:36:52.001532 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.001466    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" podStartSLOduration=6.001446144 podStartE2EDuration="6.001446144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:51.883137676 +0000 UTC m=+7.626155558" watchObservedRunningTime="2026-10-30 21:36:52.001446144 +0000 UTC m=+7.744464024"
Oct 30 21:36:52.711636 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.711540    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:52.711813 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:52.711693    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:52.712136 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.712114    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:52.712226 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:52.712210    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:52.712305 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.712290    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:52.712394 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:52.712376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:52.791176 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.791135    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerStarted","Data":"05da2be9e21af354c4fdeda6473b1e06353026bd4464fba5041a65e927d47acf"}
Oct 30 21:36:52.794809 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.794776    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-sxl4l" event={"ID":"2bf2774c-7d35-4bff-98ae-489cc598b066","Type":"ContainerStarted","Data":"797ce072906fa8d9b4e57818111ddbcbda6830229cf79fa06201cb1885845953"}
Oct 30 21:36:52.804092 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.804055    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="158a926b33cc7e282cfa6b749be03fd5902196af85b89c408d5906d631bbe6a5" exitCode=0
Oct 30 21:36:52.805151 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.805127    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"158a926b33cc7e282cfa6b749be03fd5902196af85b89c408d5906d631bbe6a5"}
Oct 30 21:36:52.911065 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:52.911007    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" podStartSLOduration=6.910988056 podStartE2EDuration="6.910988056s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:52.910180794 +0000 UTC m=+8.653198674" watchObservedRunningTime="2026-10-30 21:36:52.910988056 +0000 UTC m=+8.654005936"
Oct 30 21:36:53.152816 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:53.152717    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-sxl4l" podStartSLOduration=2.897967546 podStartE2EDuration="8.152695633s" totalImagesPullingTime="5.254728087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:53.0285434 +0000 UTC m=+8.771561280" watchObservedRunningTime="2026-10-30 21:36:53.152695633 +0000 UTC m=+8.895713515"
Oct 30 21:36:53.822079 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:53.822038    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-7rdhn" event={"ID":"f0d8c917-e87e-4304-bec9-66f213536b88","Type":"ContainerStarted","Data":"baaf2f951aeeb489ebc3899aeb248c5b35078e9a722c13da12186e16b55332a0"}
Oct 30 21:36:53.938799 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:53.937690    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-7rdhn" podStartSLOduration=2.766499334 podStartE2EDuration="8.937669203s" totalImagesPullingTime="6.171169869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:53.937484227 +0000 UTC m=+9.680502122" watchObservedRunningTime="2026-10-30 21:36:53.937669203 +0000 UTC m=+9.680687085"
Oct 30 21:36:54.420317 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.420280    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:54.420494 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.420422    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:54.420494 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.420468    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:02.420455147 +0000 UTC m=+18.163473005 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:36:54.535096 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.528010    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:54.535096 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.528286    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:54.535096 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.528361    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:02.528342673 +0000 UTC m=+18.271360534 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:36:54.629669 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.628877    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:54.629669 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.629037    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:36:54.629669 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.629059    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:36:54.629669 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.629071    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:54.629669 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.629129    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:02.629110601 +0000 UTC m=+18.372128473 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:36:54.710875 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.710303    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:54.710875 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.710421    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:54.710875 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.710676    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:54.710875 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.710778    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:54.711310 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.711201    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:54.711310 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:54.711276    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:54.828990 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:54.828956    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b" event={"ID":"79565b90-0624-4317-983b-6f44b73c6f78","Type":"ContainerStarted","Data":"69d71cb377ff8a9452f84e01d434be4d00719b3a5f15bdf46bd065f1846b915f"}
Oct 30 21:36:55.835293 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:55.834442    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z6crj" event={"ID":"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749","Type":"ContainerStarted","Data":"faa3da4fc91b5ade5abcd92f6edba79745f8ae72c794747d146534374274a3d4"}
Oct 30 21:36:56.128514 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.128484    2578 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:36:56.709628 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.709578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:56.709811 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.709578    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:56.709811 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:56.709717    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:56.709811 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.709605    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:56.709953 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:56.709852    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:56.709953 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:56.709942    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:56.716045 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.715949    2578 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:36:56.128507523Z","UUID":"d884969a-9ebc-4442-b6d5-43b9059b593b","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:36:56.717634 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.717615    2578 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:36:56.717634 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.717640    2578 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:36:56.839513 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.839466    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b" event={"ID":"79565b90-0624-4317-983b-6f44b73c6f78","Type":"ContainerStarted","Data":"ca0309c866ad5bb8e6840d88b838b1a548ac63264526161d79aabb4ee6ad0c04"}
Oct 30 21:36:56.841877 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.841848    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-4cmqj" event={"ID":"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79","Type":"ContainerStarted","Data":"d99f8433c43f4df7401f58a52369088ac77e1fa7839ffb4d722c6f349f1bb1c1"}
Oct 30 21:36:56.958696 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.958426    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-z6crj" podStartSLOduration=3.36538295 podStartE2EDuration="11.958407999s" totalImagesPullingTime="8.593025049s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:55.955440426 +0000 UTC m=+11.698458306" watchObservedRunningTime="2026-10-30 21:36:56.958407999 +0000 UTC m=+12.701425884"
Oct 30 21:36:56.958890 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:56.958855    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-4cmqj" podStartSLOduration=2.914810354 podStartE2EDuration="11.958846368s" totalImagesPullingTime="9.044036014s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:56.958087212 +0000 UTC m=+12.701105091" watchObservedRunningTime="2026-10-30 21:36:56.958846368 +0000 UTC m=+12.701864249"
Oct 30 21:36:57.394793 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:57.394757    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:57.395638 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:57.395611    2578 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:36:57.846957 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:57.846921    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="73476c3b26aeb4f738d57aad80ab944a5e00be5e522c756153ce8d99c1095b1e" exitCode=0
Oct 30 21:36:57.847436 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:57.847031    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"73476c3b26aeb4f738d57aad80ab944a5e00be5e522c756153ce8d99c1095b1e"}
Oct 30 21:36:57.849293 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:57.849196    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-pz6fg" event={"ID":"8ee90e5a-7c9f-4b24-841c-86b50175efd6","Type":"ContainerStarted","Data":"a8426a71b09c79b15b561fb39d7143d0b6e137471ac4b4ee6d8259b2bd0c3f78"}
Oct 30 21:36:58.138743 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.138682    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-pz6fg" podStartSLOduration=3.939364483 podStartE2EDuration="13.138662594s" totalImagesPullingTime="9.199298111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:58.138244444 +0000 UTC m=+13.881262324" watchObservedRunningTime="2026-10-30 21:36:58.138662594 +0000 UTC m=+13.881680474"
Oct 30 21:36:58.709298 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.709279    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:36:58.709298 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.709291    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:36:58.709447 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:58.709376    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:36:58.709479 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:58.709462    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:36:58.709528 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.709517    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:36:58.709598 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:36:58.709574    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:36:58.853148 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.853107    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-fgd96" event={"ID":"e021af5f-e7c2-474e-aff2-7cc532fbcaf0","Type":"ContainerStarted","Data":"28b62ccf4945a0266ba47f3ca170d2fffe25c1c2311897f19ff94177b70d8b5f"}
Oct 30 21:36:58.858941 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.858912    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b" event={"ID":"79565b90-0624-4317-983b-6f44b73c6f78","Type":"ContainerStarted","Data":"e1254890db676ed6a97e389dadf788ace2c559b9cea5a7272fd81730a275d195"}
Oct 30 21:36:58.972976 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:58.972870    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-fgd96" podStartSLOduration=2.408090074 podStartE2EDuration="13.972856211s" totalImagesPullingTime="11.564766137s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:58.97216635 +0000 UTC m=+14.715184231" watchObservedRunningTime="2026-10-30 21:36:58.972856211 +0000 UTC m=+14.715874090"
Oct 30 21:36:59.863279 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:59.863246    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="c581dd26c2fd543e61e1bd25b479a44a2af40106628a620953c6b2ca37cab3e5" exitCode=0
Oct 30 21:36:59.863931 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:59.863334    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"c581dd26c2fd543e61e1bd25b479a44a2af40106628a620953c6b2ca37cab3e5"}
Oct 30 21:36:59.986639 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:36:59.986562    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b" podStartSLOduration=4.097580302 podStartE2EDuration="14.986541165s" totalImagesPullingTime="10.888960863s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:36:59.093063891 +0000 UTC m=+14.836081773" watchObservedRunningTime="2026-10-30 21:36:59.986541165 +0000 UTC m=+15.729559046"
Oct 30 21:37:00.708918 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:00.708876    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:00.708918 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:00.708903    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:00.709100 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:00.708877    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:00.709100 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:00.709000    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:00.709100 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:00.709062    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:00.709223 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:00.709120    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:01.871088 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:01.870686    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="4e3c1cae641e35e914277aaa560a5d504948557f13798cd58869793a1e21b0fe" exitCode=0
Oct 30 21:37:01.871088 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:01.870774    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"4e3c1cae641e35e914277aaa560a5d504948557f13798cd58869793a1e21b0fe"}
Oct 30 21:37:02.493382 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.493342    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:02.493544 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.493513    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:37:02.493628 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.493577    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:18.493561765 +0000 UTC m=+34.236579623 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:37:02.593946 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.593915    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:02.594146 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.594039    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:37:02.594146 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.594105    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:18.594088171 +0000 UTC m=+34.337106043 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:37:02.695157 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.695121    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:02.695307 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.695288    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:37:02.695342 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.695312    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:37:02.695342 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.695323    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:37:02.695422 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.695376    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:18.695358899 +0000 UTC m=+34.438376774 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:37:02.709353 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.709328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:02.709417 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.709350    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:02.709417 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:02.709385    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:02.709489 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.709461    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:02.709536 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.709521    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:02.709640 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:02.709619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:03.272630 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.272405    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:03.273007 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.272727    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:37:03.273170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.273149    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:03.879515 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879464    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"0f021550ef4af720a9549abd26ab22701b20e625e63042d0d52bf7259504250b"}
Oct 30 21:37:03.879737 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879571    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"6aff11f7ccab0de69437e97be0eebbc39cfc8cb428e545a5a9540903d40d76ed"}
Oct 30 21:37:03.879737 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879628    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"875dc33503f5fd2b555301d84721e8ddef0f9944188d997775c0d4ac1ecb4a2e"}
Oct 30 21:37:03.879737 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879679    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"32d4751d9903731f92952fad0d68f9a660e57f880c0f7eb6302a8432cdba7acf"}
Oct 30 21:37:03.879737 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879717    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"4fef45c88cb6006a469c44c0590809e82e306acf249ba81bec38fa43ab2f5831"}
Oct 30 21:37:03.879915 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:03.879750    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"8db4e7fe8e437484a1c7fe5bb928b439a2189bb81868eaa767ff99f761b97125"}
Oct 30 21:37:04.709541 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:04.709506    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:04.710174 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:04.709632    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:04.710174 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:04.709698    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:04.710174 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:04.709807    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:04.710174 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:04.709861    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:04.713099 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:04.710392    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:04.884523 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:04.884490    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="17b3886ad1259fabadd023c7f6a524ed08e2832b787090dec0902ceb0355394f" exitCode=0
Oct 30 21:37:04.884693 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:04.884555    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"17b3886ad1259fabadd023c7f6a524ed08e2832b787090dec0902ceb0355394f"}
Oct 30 21:37:05.889783 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:05.889736    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"4a9f10e29a5fe7a3fbd375028df3b6b3aab10b17ed0ac210c90db7b825ecc4ef"}
Oct 30 21:37:05.892724 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:05.892698    2578 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="3d14f8003915a8917e146b42d6f7a7a5e06285f845b9603567aa39518f0e23aa" exitCode=0
Oct 30 21:37:05.892839 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:05.892757    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerDied","Data":"3d14f8003915a8917e146b42d6f7a7a5e06285f845b9603567aa39518f0e23aa"}
Oct 30 21:37:06.709602 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:06.709544    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:06.709779 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:06.709562    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:06.709779 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:06.709682    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:06.709847 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:06.709814    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:06.709847 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:06.709841    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:06.709921 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:06.709905    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:06.898339 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:06.898299    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" event={"ID":"07f98edf-bb0a-422e-ab8f-c1f935774f2a","Type":"ContainerStarted","Data":"7ced10042c1370d5152b139227b9e4de3868a70baf6379aca11ac09e149f02d0"}
Oct 30 21:37:07.073498 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:07.073394    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-k4kwl" podStartSLOduration=8.852190864 podStartE2EDuration="22.073380066s" totalImagesPullingTime="13.221189202s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:07.0731312 +0000 UTC m=+22.816149081" watchObservedRunningTime="2026-10-30 21:37:07.073380066 +0000 UTC m=+22.816397945"
Oct 30 21:37:07.904417 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:07.904206    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" event={"ID":"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa","Type":"ContainerStarted","Data":"c6a0acd3e0153b2e4066e98b29b161b937d52a18bf8fb6d6d3152bff5bbf9623"}
Oct 30 21:37:08.185948 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.185848    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf" podStartSLOduration=7.458519552 podStartE2EDuration="23.185833538s" totalImagesPullingTime="15.727313986s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:08.185447119 +0000 UTC m=+23.928464996" watchObservedRunningTime="2026-10-30 21:37:08.185833538 +0000 UTC m=+23.928851418"
Oct 30 21:37:08.709258 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.709217    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:08.709423 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.709271    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:08.709423 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.709228    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:08.709423 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:08.709353    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:08.709533 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:08.709471    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:08.709597 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:08.709566    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:08.906808 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.906780    2578 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:37:08.907212 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.907182    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:08.907349 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.907257    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:08.922843 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.922811    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:08.922987 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:08.922917    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:09.120368 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.120336    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:09.947288 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.947246    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6pwrc"]
Oct 30 21:37:09.947770 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.947421    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:09.947770 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:09.947539    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:09.961377 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.961351    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x465r"]
Oct 30 21:37:09.961520 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.961464    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:09.961562 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:09.961546    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:09.964124 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.964105    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdqv5"]
Oct 30 21:37:09.964203 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:09.964190    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:09.964281 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:09.964266    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:11.709289 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:11.709202    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:11.709709 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:11.709210    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:11.709709 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:11.709328    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:11.709709 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:11.709411    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:11.709709 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:11.709219    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:11.709709 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:11.709527    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:13.710234 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:13.710193    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:13.710779 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:13.710273    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:13.710779 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:13.710375    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:13.710779 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:13.710408    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:13.710779 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:13.710467    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:13.710779 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:13.710503    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:15.709771 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:15.709556    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:15.710169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:15.709617    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:15.710169 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:15.709810    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:15.710169 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:15.709628    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:15.710169 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:15.709938    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:15.710169 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:15.709998    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:17.709309 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:17.709272    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:17.709309 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:17.709309    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:17.709766 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:17.709440    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:17.709766 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:17.709483    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:17.709766 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:17.709558    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:17.709766 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:17.709643    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:18.503209 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:18.503161    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:18.503401 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.503312    2578 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:37:18.503401 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.503381    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs podName:f1d1b936-7975-413e-8b25-ace2bd71cdd3 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:50.50336553 +0000 UTC m=+66.246383388 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs") pod "network-metrics-daemon-x465r" (UID: "f1d1b936-7975-413e-8b25-ace2bd71cdd3") : object "openshift-multus"/"metrics-daemon-secret" not registered
Oct 30 21:37:18.604111 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:18.604071    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:18.604283 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.604224    2578 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Oct 30 21:37:18.604321 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.604290    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret podName:d97ac33e-24e8-4256-a3cd-154231e10553 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:50.604273428 +0000 UTC m=+66.347291286 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret") pod "global-pull-secret-syncer-6pwrc" (UID: "d97ac33e-24e8-4256-a3cd-154231e10553") : object "kube-system"/"original-pull-secret" not registered
Oct 30 21:37:18.704735 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:18.704698    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:18.704899 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.704861    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Oct 30 21:37:18.704899 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.704883    2578 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Oct 30 21:37:18.704899 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.704893    2578 projected.go:196] Error preparing data for projected volume kube-api-access-vgn88 for pod openshift-network-diagnostics/network-check-target-fdqv5: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:37:18.704997 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:18.704949    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88 podName:3b4f2e52-d72d-4962-9627-e1817b1774a9 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:50.704936184 +0000 UTC m=+66.447954069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-vgn88" (UniqueName: "kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88") pod "network-check-target-fdqv5" (UID: "3b4f2e52-d72d-4962-9627-e1817b1774a9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Oct 30 21:37:19.709419 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:19.709330    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:19.709931 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:19.709334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:19.709931 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:19.709458    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:19.709931 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:19.709334    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:19.709931 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:19.709527    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:19.709931 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:19.709619    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:21.708836 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:21.708793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:21.708836 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:21.708836    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:21.709260 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:21.708793    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:21.709260 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:21.708925    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:21.709260 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:21.709015    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:21.709260 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:21.709089    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:23.708656 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:23.708626    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:23.709133 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:23.708627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:23.709133 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:23.708627    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:23.709133 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:23.708828    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:23.709133 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:23.708716    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:23.709133 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:23.708881    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:25.709046 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:25.709007    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:25.709046 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:25.709013    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:25.709460 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:25.709187    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-6pwrc" podUID="d97ac33e-24e8-4256-a3cd-154231e10553"
Oct 30 21:37:25.709460 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:25.709213    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:25.709460 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:25.709285    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-x465r" podUID="f1d1b936-7975-413e-8b25-ace2bd71cdd3"
Oct 30 21:37:25.709460 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:25.709330    2578 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fdqv5" podUID="3b4f2e52-d72d-4962-9627-e1817b1774a9"
Oct 30 21:37:26.543257 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.543178    2578 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:37:26.543420 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.543325    2578 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Oct 30 21:37:26.670488 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.670455    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-b7bx8"]
Oct 30 21:37:26.692562 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.692532    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.717699 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.717676    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-6cj9n"]
Oct 30 21:37:26.730545 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.730515    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7bx8"]
Oct 30 21:37:26.730699 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.730663    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:26.746730 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.746701    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6cj9n"]
Oct 30 21:37:26.799988 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.799906    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:37:26.799988 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.799915    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:37:26.799988 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.799914    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:37:26.800229 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.800019    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:37:26.800229 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.800056    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:37:26.800229 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.800091    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-phtlh\""
Oct 30 21:37:26.800229 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.800098    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb4gr\""
Oct 30 21:37:26.860650 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860611    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ggqtk\" (UniqueName: \"kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.860850 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860666    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:26.860850 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860791    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h7pqv\" (UniqueName: \"kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:26.860850 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860824    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.860963 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860928    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.860963 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.860950    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.961996 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.961960    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:26.962170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962013    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pqv\" (UniqueName: \"kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:26.962170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962033    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.962170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962074    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.962170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962090    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.962170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962112    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqtk\" (UniqueName: \"kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.962411 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:26.962222    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:37:26.962411 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:26.962332    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls podName:ef440513-6ad3-4663-92fb-1b69c378c66f nodeName:}" failed. No retries permitted until 2026-10-30 21:37:27.462308234 +0000 UTC m=+43.205326116 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls") pod "dns-default-b7bx8" (UID: "ef440513-6ad3-4663-92fb-1b69c378c66f") : secret "dns-default-metrics-tls" not found
Oct 30 21:37:26.962490 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962450    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.962718 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.962701    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:26.965955 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:26.965936    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:27.026268 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.026233    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqtk\" (UniqueName: \"kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:27.026413 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.026330    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pqv\" (UniqueName: \"kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:27.056283 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.056196    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:27.465671 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.465635    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:27.465856 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:27.465837    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:37:27.465918 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:27.465907    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls podName:ef440513-6ad3-4663-92fb-1b69c378c66f nodeName:}" failed. No retries permitted until 2026-10-30 21:37:28.465890103 +0000 UTC m=+44.208907987 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls") pod "dns-default-b7bx8" (UID: "ef440513-6ad3-4663-92fb-1b69c378c66f") : secret "dns-default-metrics-tls" not found
Oct 30 21:37:27.492675 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.492641    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-6cj9n"]
Oct 30 21:37:27.492791 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:27.492721    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1a38f40_6f73_4d05_9ca4_d82518e4bbe7.slice/crio-a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb WatchSource:0}: Error finding container a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb: Status 404 returned error can't find the container with id a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb
Oct 30 21:37:27.709236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.709147    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:27.709236 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.709170    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:27.709447 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.709343    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:27.763310 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763273    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\""
Oct 30 21:37:27.763310 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763288    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ktl9g\""
Oct 30 21:37:27.763785 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763320    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:37:27.763785 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763347    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\""
Oct 30 21:37:27.763785 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763605    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:37:27.763785 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763667    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:37:27.763785 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.763726    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:37:27.949230 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:27.949191    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6cj9n" event={"ID":"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7","Type":"ContainerStarted","Data":"a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb"}
Oct 30 21:37:28.472761 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:28.472723    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:28.472910 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:28.472844    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:37:28.472910 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:28.472896    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls podName:ef440513-6ad3-4663-92fb-1b69c378c66f nodeName:}" failed. No retries permitted until 2026-10-30 21:37:30.472883022 +0000 UTC m=+46.215900879 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls") pod "dns-default-b7bx8" (UID: "ef440513-6ad3-4663-92fb-1b69c378c66f") : secret "dns-default-metrics-tls" not found
Oct 30 21:37:30.487994 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:30.487950    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:30.488507 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:30.488117    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:37:30.488507 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:30.488188    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls podName:ef440513-6ad3-4663-92fb-1b69c378c66f nodeName:}" failed. No retries permitted until 2026-10-30 21:37:34.48817273 +0000 UTC m=+50.231190588 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls") pod "dns-default-b7bx8" (UID: "ef440513-6ad3-4663-92fb-1b69c378c66f") : secret "dns-default-metrics-tls" not found
Oct 30 21:37:31.960543 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:31.960507    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-6cj9n" event={"ID":"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7","Type":"ContainerStarted","Data":"70add7f8fdf2097d4571fdaa49ad03bf047dcaa7d22c2c4b8c523c2121232dee"}
Oct 30 21:37:32.078852 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:32.078803    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-6cj9n" podStartSLOduration=2.58068812 podStartE2EDuration="6.078787573s" totalImagesPullingTime="3.498099453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:32.07839876 +0000 UTC m=+47.821416640" watchObservedRunningTime="2026-10-30 21:37:32.078787573 +0000 UTC m=+47.821805454"
Oct 30 21:37:33.676031 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.676002    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-p2prz"]
Oct 30 21:37:33.677987 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.677971    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.731770 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.731737    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:37:33.782251 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.782206    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlcqp\""
Oct 30 21:37:33.782251 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.782230    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:37:33.782251 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.782251    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:37:33.782560 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.782292    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:37:33.783170 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.783147    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2prz"]
Oct 30 21:37:33.816140 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.816104    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.816287 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.816150    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvgsw\" (UniqueName: \"kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.816287 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.816195    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.816287 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.816259    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.816385 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.816287    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.917560 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917517    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.917802 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917577    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgsw\" (UniqueName: \"kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.917802 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917648    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.917802 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917677    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.917802 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917704    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:33.917835    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917839    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:33.917849    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:33.917906    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:34.417887288 +0000 UTC m=+50.160905168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : configmap "kube-rbac-proxy" not found
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:33.917925    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:34.417915358 +0000 UTC m=+50.160933223 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:33.918003 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.917953    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:33.942995 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.942930    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:37:33.944753 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.944739    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:33.983801 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:33.983768    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgsw\" (UniqueName: \"kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:34.001965 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.001927    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4rnkh\""
Oct 30 21:37:34.054287 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.054252    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:37:34.078338 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.078302    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:37:34.078542 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.078364    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:37:34.119917 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.119887    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.120023 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.119932    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.120023 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.119951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.120104 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.120068    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221209 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221136    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221209 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221172    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221376 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221212    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221376 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221275    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221725 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221703    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.221900 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.221879    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.222148 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.222128    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.283567 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.283529    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.423388 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.423332    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:34.423388 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.423391    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:34.423630 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.423476    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:34.423630 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.423525    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:37:34.423630 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.423540    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:35.423526818 +0000 UTC m=+51.166544675 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:34.423630 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.423601    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:35.42357136 +0000 UTC m=+51.166589222 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : configmap "kube-rbac-proxy" not found
Oct 30 21:37:34.524520 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.524435    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:34.524676 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.524582    2578 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Oct 30 21:37:34.524676 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:34.524672    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls podName:ef440513-6ad3-4663-92fb-1b69c378c66f nodeName:}" failed. No retries permitted until 2026-10-30 21:37:42.524655838 +0000 UTC m=+58.267673722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls") pod "dns-default-b7bx8" (UID: "ef440513-6ad3-4663-92fb-1b69c378c66f") : secret "dns-default-metrics-tls" not found
Oct 30 21:37:34.554345 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.554316    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:34.917398 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:34.917320    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3de0e99_5c49_4347_878a_58dbd3fd404b.slice/crio-ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa WatchSource:0}: Error finding container ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa: Status 404 returned error can't find the container with id ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa
Oct 30 21:37:34.918859 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.918836    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:37:34.968671 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:34.968639    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sw5w-75z9c" event={"ID":"f3de0e99-5c49-4347-878a-58dbd3fd404b","Type":"ContainerStarted","Data":"ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa"}
Oct 30 21:37:35.277119 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.277036    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"]
Oct 30 21:37:35.279786 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.279767    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:35.336694 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.336650    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:37:35.336694 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.336667    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2fnrw\""
Oct 30 21:37:35.336907 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.336741    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:37:35.337026 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.337006    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"]
Oct 30 21:37:35.430986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.430951    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfsf7\" (UniqueName: \"kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7\") pod \"migrator-66df5d496b-6n6ft\" (UID: \"2328abe6-8360-4e86-9a59-4e308704aac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:35.430986 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.431002    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:35.431241 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.431045    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:35.431241 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:35.431104    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:35.431241 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:35.431162    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:37.4311496 +0000 UTC m=+53.174167458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:35.431241 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:35.431195    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:37:35.431391 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:35.431254    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:37.431236523 +0000 UTC m=+53.174254384 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : configmap "kube-rbac-proxy" not found
Oct 30 21:37:35.532141 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.532065    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsf7\" (UniqueName: \"kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7\") pod \"migrator-66df5d496b-6n6ft\" (UID: \"2328abe6-8360-4e86-9a59-4e308704aac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:35.593281 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.593248    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsf7\" (UniqueName: \"kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7\") pod \"migrator-66df5d496b-6n6ft\" (UID: \"2328abe6-8360-4e86-9a59-4e308704aac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:35.890980 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:35.890940    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:36.042342 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.042302    2578 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vrzl6"]
Oct 30 21:37:36.045354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.045328    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.099823 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.099776    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:37:36.099823 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.099816    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nc2jc\""
Oct 30 21:37:36.100154 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.100114    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:37:36.100267 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.100211    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:37:36.100987 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.100959    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vrzl6"]
Oct 30 21:37:36.136782 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.136720    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.136984 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.136877    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.137050 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.136983    2578 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqjqw\" (UniqueName: \"kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.200421 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.200334    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:37:36.237709 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.237664    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.237914 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.237789    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.237914 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.237851    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjqw\" (UniqueName: \"kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.238645 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.238617    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.240692 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.240662    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.271680 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:36.271646    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2328abe6_8360_4e86_9a59_4e308704aac1.slice/crio-bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d WatchSource:0}: Error finding container bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d: Status 404 returned error can't find the container with id bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d
Oct 30 21:37:36.272757 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.272730    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"]
Oct 30 21:37:36.300316 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.300287    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjqw\" (UniqueName: \"kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.358376 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.358342    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:36.735453 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.735420    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-5fffc97bdf-vrzl6"]
Oct 30 21:37:36.736015 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:36.735968    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5da833b_0939_4f20_a2b3_18fe731a9360.slice/crio-3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b WatchSource:0}: Error finding container 3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b: Status 404 returned error can't find the container with id 3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b
Oct 30 21:37:36.975551 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.975514    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6" event={"ID":"b5da833b-0939-4f20-a2b3-18fe731a9360","Type":"ContainerStarted","Data":"3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b"}
Oct 30 21:37:36.976923 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:36.976886    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft" event={"ID":"2328abe6-8360-4e86-9a59-4e308704aac1","Type":"ContainerStarted","Data":"bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d"}
Oct 30 21:37:37.446711 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:37.446683    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:37.447119 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:37.446719    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:37.447119 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:37.446815    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:37:37.447119 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:37.446844    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:37.447119 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:37.446879    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:41.446861401 +0000 UTC m=+57.189879273 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : configmap "kube-rbac-proxy" not found
Oct 30 21:37:37.447119 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:37.446897    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:41.446890994 +0000 UTC m=+57.189908851 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:37.981853 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:37.981812    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sw5w-75z9c" event={"ID":"f3de0e99-5c49-4347-878a-58dbd3fd404b","Type":"ContainerStarted","Data":"9ce770983b879bc3f104f1e576b97c68ab4337aa2a8da56dc42cdba4bcc3798c"}
Oct 30 21:37:38.151698 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:38.151638    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-7sw5w-75z9c" podStartSLOduration=2.668909002 podStartE2EDuration="5.151617958s" totalImagesPullingTime="2.482708956s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:38.151564965 +0000 UTC m=+53.894582845" watchObservedRunningTime="2026-10-30 21:37:38.151617958 +0000 UTC m=+53.894635836"
Oct 30 21:37:38.986416 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:38.986375    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft" event={"ID":"2328abe6-8360-4e86-9a59-4e308704aac1","Type":"ContainerStarted","Data":"f0d3d25ccb67254a27956d7185b115e1dd11001773aeebaae0613ef486b7888f"}
Oct 30 21:37:39.108860 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:39.108809    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft" podStartSLOduration=2.4346959 podStartE2EDuration="4.10879243s" totalImagesPullingTime="1.67409653s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:39.107767064 +0000 UTC m=+54.850784942" watchObservedRunningTime="2026-10-30 21:37:39.10879243 +0000 UTC m=+54.851810308"
Oct 30 21:37:39.990301 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:39.990208    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6" event={"ID":"b5da833b-0939-4f20-a2b3-18fe731a9360","Type":"ContainerStarted","Data":"ca9f985c7ab418e6a603da8c5600da4d3f673d39255e392a3029c896af1f3d40"}
Oct 30 21:37:40.108382 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:40.108315    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6" podStartSLOduration=2.190889343 podStartE2EDuration="5.108294279s" totalImagesPullingTime="2.917404936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:40.107524266 +0000 UTC m=+55.850542145" watchObservedRunningTime="2026-10-30 21:37:40.108294279 +0000 UTC m=+55.851312162"
Oct 30 21:37:40.922398 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:40.922363    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:41.217466 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:41.217394    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-4cmqj_4d6a00c1-5c3d-48e9-974d-9ac8477c2f79/dns-node-resolver/0.log"
Oct 30 21:37:41.480373 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:41.480289    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:41.480373 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:41.480329    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:41.480564 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:41.480405    2578 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Oct 30 21:37:41.480564 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:41.480446    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:41.480564 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:41.480457    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:49.480444814 +0000 UTC m=+65.223462683 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : configmap "kube-rbac-proxy" not found
Oct 30 21:37:41.480564 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:41.480509    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:37:49.480495594 +0000 UTC m=+65.223513452 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:42.223076 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:42.223043    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-7rdhn_f0d8c917-e87e-4304-bec9-66f213536b88/node-ca/0.log"
Oct 30 21:37:42.587945 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:42.587911    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:42.590197 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:42.590167    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:42.631371 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:42.631332    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:42.998908 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:42.998868    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef440513_6ad3_4663_92fb_1b69c378c66f.slice/crio-1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2 WatchSource:0}: Error finding container 1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2: Status 404 returned error can't find the container with id 1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2
Oct 30 21:37:43.004779 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:43.004708    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-b7bx8"]
Oct 30 21:37:43.223997 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:43.223938    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-6cj9n_a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/serve-healthcheck-canary/0.log"
Oct 30 21:37:43.813538 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:43.813490    2578 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-6n6ft_2328abe6-8360-4e86-9a59-4e308704aac1/migrator/0.log"
Oct 30 21:37:44.000951 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:44.000915    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7bx8" event={"ID":"ef440513-6ad3-4663-92fb-1b69c378c66f","Type":"ContainerStarted","Data":"1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2"}
Oct 30 21:37:46.008499 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:46.008444    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7bx8" event={"ID":"ef440513-6ad3-4663-92fb-1b69c378c66f","Type":"ContainerStarted","Data":"8b50614eb86951635197a2f7fb67e0fbc38a19a5507a2464d9fce35669ed0196"}
Oct 30 21:37:46.008925 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:46.008558    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-b7bx8" event={"ID":"ef440513-6ad3-4663-92fb-1b69c378c66f","Type":"ContainerStarted","Data":"307c9727c8bcd5fc562728bb0451a9c35513e55650172ed23ec0c0192d61a3b1"}
Oct 30 21:37:46.008925 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:46.008613    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:46.128152 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:46.128096    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-b7bx8" podStartSLOduration=17.925795016 podStartE2EDuration="20.12808145s" totalImagesPullingTime="2.202286434s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:37:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:46.12751014 +0000 UTC m=+61.870528020" watchObservedRunningTime="2026-10-30 21:37:46.12808145 +0000 UTC m=+61.871099351"
Oct 30 21:37:49.542798 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:49.542765    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:49.542798 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:49.542803    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:49.543215 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:49.542918    2578 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:49.543215 ip-10-0-141-82 kubenswrapper[2578]: E1030 21:37:49.542982    2578 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls podName:abb604c9-145e-4bdb-8679-a5a0c25b1441 nodeName:}" failed. No retries permitted until 2026-10-30 21:38:05.542967761 +0000 UTC m=+81.285985624 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls") pod "insights-runtime-extractor-p2prz" (UID: "abb604c9-145e-4bdb-8679-a5a0c25b1441") : secret "insights-runtime-extractor-tls" not found
Oct 30 21:37:49.543354 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:49.543338    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:50.550545 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.550499    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:50.603073 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.603038    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:37:50.613333 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.613301    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:50.651676 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.651640    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:50.704240 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.704208    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:37:50.713874 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.713845    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:50.752398 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.752359    2578 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:50.804883 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.804812    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:37:50.865197 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.865167    2578 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:37:50.875943 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.875908    2578 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:50.893249 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.893225    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\""
Oct 30 21:37:50.901440 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.901422    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:50.920835 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.920805    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\""
Oct 30 21:37:50.928980 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.928946    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:50.935343 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.935317    2578 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ktl9g\""
Oct 30 21:37:50.943656 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:50.943625    2578 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:51.280198 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:51.278997    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-6pwrc"]
Oct 30 21:37:51.307794 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:51.307756    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97ac33e_24e8_4256_a3cd_154231e10553.slice/crio-eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3 WatchSource:0}: Error finding container eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3: Status 404 returned error can't find the container with id eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3
Oct 30 21:37:51.320327 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:51.320301    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1d1b936_7975_413e_8b25_ace2bd71cdd3.slice/crio-797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d WatchSource:0}: Error finding container 797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d: Status 404 returned error can't find the container with id 797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d
Oct 30 21:37:51.321449 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:51.321424    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-x465r"]
Oct 30 21:37:51.332010 ip-10-0-141-82 kubenswrapper[2578]: W1030 21:37:51.331985    2578 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4f2e52_d72d_4962_9627_e1817b1774a9.slice/crio-72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90 WatchSource:0}: Error finding container 72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90: Status 404 returned error can't find the container with id 72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90
Oct 30 21:37:51.333128 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:51.333107    2578 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fdqv5"]
Oct 30 21:37:52.025566 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:52.025532    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdqv5" event={"ID":"3b4f2e52-d72d-4962-9627-e1817b1774a9","Type":"ContainerStarted","Data":"72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90"}
Oct 30 21:37:52.026848 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:52.026818    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-6pwrc" event={"ID":"d97ac33e-24e8-4256-a3cd-154231e10553","Type":"ContainerStarted","Data":"eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3"}
Oct 30 21:37:52.027871 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:52.027850    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x465r" event={"ID":"f1d1b936-7975-413e-8b25-ace2bd71cdd3","Type":"ContainerStarted","Data":"797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d"}
Oct 30 21:37:54.037851 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:54.037191    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x465r" event={"ID":"f1d1b936-7975-413e-8b25-ace2bd71cdd3","Type":"ContainerStarted","Data":"a0283b7e58f5f9608aa830e1e99d8d4346bf45b181cb1c1280d70b0bb972e2ae"}
Oct 30 21:37:54.037851 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:54.037259    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-x465r" event={"ID":"f1d1b936-7975-413e-8b25-ace2bd71cdd3","Type":"ContainerStarted","Data":"8f293a6daf5b1223bf54a6d94e58c6e9bd9074a4c73ab961bd6193bbe5aefd9f"}
Oct 30 21:37:54.156912 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:54.156844    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-x465r" podStartSLOduration=67.425828862 podStartE2EDuration="1m9.156822354s" totalImagesPullingTime="1.730993492s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:54.156010719 +0000 UTC m=+69.899028597" watchObservedRunningTime="2026-10-30 21:37:54.156822354 +0000 UTC m=+69.899840234"
Oct 30 21:37:56.013715 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:56.013682    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:56.047347 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:56.047304    2578 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fdqv5" event={"ID":"3b4f2e52-d72d-4962-9627-e1817b1774a9","Type":"ContainerStarted","Data":"2aedc63651a61fbd149805cd94e119ccba74221660a5b1a4202e86fb3254aede"}
Oct 30 21:37:56.047522 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:56.047440    2578 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:56.258602 ip-10-0-141-82 kubenswrapper[2578]: I1030 21:37:56.258547    2578 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fdqv5" podStartSLOduration=67.438131026 podStartE2EDuration="1m11.258530847s" totalImagesPullingTime="3.820399821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:36:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:37:56.258522155 +0000 UTC m=+72.001540036" watchObservedRunningTime="2026-10-30 21:37:56.258530847 +0000 UTC m=+72.001548728"
Oct 30 21:37:56.770082 ip-10-0-141-82 systemd[1]: Stopping Kubernetes Kubelet...
Oct 30 21:37:56.784969 ip-10-0-141-82 systemd[1]: kubelet.service: Deactivated successfully.
Oct 30 21:37:56.785134 ip-10-0-141-82 systemd[1]: Stopped Kubernetes Kubelet.
Oct 30 21:37:56.785182 ip-10-0-141-82 systemd[1]: kubelet.service: Consumed 3.460s CPU time.
Oct 30 21:37:56.796234 ip-10-0-141-82 systemd[1]: Starting Kubernetes Kubelet...
Oct 30 21:37:56.857601 ip-10-0-141-82 kubenswrapper[4882]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:37:56.857601 ip-10-0-141-82 kubenswrapper[4882]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Oct 30 21:37:56.857601 ip-10-0-141-82 kubenswrapper[4882]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:37:56.857601 ip-10-0-141-82 kubenswrapper[4882]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.859959    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860053    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860057    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860061    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860065    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.860059 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860069    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860073    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860077    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860081    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860085    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860088    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860096    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860102    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860106    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860110    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860114    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860117    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860122    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860125    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860129    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860133    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860147    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860151    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860154    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.860434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860158    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860162    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860166    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860170    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860174    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860177    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860183    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860188    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860193    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860198    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860202    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860206    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860211    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860215    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860219    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860225    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860229    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860234    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860238    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.861239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860242    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860247    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860251    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860255    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860260    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860265    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860269    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860273    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860277    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860281    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860286    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860290    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860294    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860298    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860302    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860306    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860311    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860315    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860319    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860323    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.862115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860327    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860332    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860338    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860342    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860346    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860351    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860355    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860360    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860364    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860370    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860374    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860378    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860382    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860386    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860391    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860395    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860400    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860404    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860409    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.863010 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860413    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860417    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860421    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860426    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860429    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860433    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860438    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860442    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860446    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860451    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860455    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860459    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860463    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860468    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860472    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860477    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860481    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860486    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860491    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860495    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.863845 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860499    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860504    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860508    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.860515    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861878    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861891    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861897    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861902    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861907    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861913    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861917    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861922    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861926    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861930    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861934    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861938    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861943    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861947    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861951    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861956    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.864737 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861960    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861965    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861969    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861973    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861982    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861986    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861991    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.861995    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862000    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862004    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862009    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862013    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862017    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862021    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862025    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862030    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862035    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862039    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862044    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.865676 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862048    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862052    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862057    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862061    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862065    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862069    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862073    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862078    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862082    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862087    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862091    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862095    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862100    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862104    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862108    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862112    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862116    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862124    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862129    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862133    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.866687 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862138    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862142    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862147    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862151    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862156    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862160    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862165    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862169    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862173    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862178    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862183    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862187    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862194    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862200    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862204    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862217    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862222    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862226    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862230    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.867239 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862235    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862239    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862244    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862248    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862252    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862257    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862261    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862265    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862272    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862277    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862285    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862290    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862295    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862300    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862304    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862309    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862313    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862318    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862322    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.867742 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862327    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862331    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862336    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862340    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862344    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862349    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862354    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862358    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862362    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862367    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862371    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862377    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.862382    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863247    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863257    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863262    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863266    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863271    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863275    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863279    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.868250 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863283    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863287    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863291    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863298    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863303    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863307    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863311    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863316    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863320    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863324    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863329    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863333    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863337    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863341    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863345    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863350    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863354    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863359    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863363    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863368    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.868816 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863372    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863376    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863380    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863384    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863388    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863392    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863396    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863401    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863405    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863409    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863413    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863417    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863421    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863425    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863430    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863436    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863440    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863444    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863448    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863454    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.869337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863460    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863464    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863468    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863472    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863476    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863481    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863485    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863489    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863493    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863497    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863502    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863506    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863510    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863515    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863519    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863523    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863527    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863531    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863535    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.869871 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863540    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863544    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863548    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863551    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863556    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863560    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863565    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863569    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863575    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863579    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863601    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863606    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863610    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863614    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863618    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863623    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863627    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863631    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863635    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.870352 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863639    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863644    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863648    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863652    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863658    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863671    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863675    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863679    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863684    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863688    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863695    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863702    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863707    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863712    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863716    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863722    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863726    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863730    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863735    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.870897 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863739    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.863745    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863893    4882 flags.go:64] FLAG: --address="0.0.0.0"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863904    4882 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863922    4882 flags.go:64] FLAG: --anonymous-auth="true"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863929    4882 flags.go:64] FLAG: --application-metrics-count-limit="100"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863936    4882 flags.go:64] FLAG: --authentication-token-webhook="false"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863941    4882 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863948    4882 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863955    4882 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863960    4882 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863965    4882 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863971    4882 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863977    4882 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863983    4882 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863987    4882 flags.go:64] FLAG: --cgroup-root=""
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863992    4882 flags.go:64] FLAG: --cgroups-per-qos="true"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.863997    4882 flags.go:64] FLAG: --client-ca-file=""
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864001    4882 flags.go:64] FLAG: --cloud-provider="external"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864006    4882 flags.go:64] FLAG: --cluster-dns="[]"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864017    4882 flags.go:64] FLAG: --cluster-domain=""
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864022    4882 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864027    4882 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864032    4882 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Oct 30 21:37:56.871535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864037    4882 flags.go:64] FLAG: --container-log-max-files="5"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864044    4882 flags.go:64] FLAG: --container-log-max-size="10Mi"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864048    4882 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864053    4882 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864058    4882 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864063    4882 flags.go:64] FLAG: --contention-profiling="false"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864071    4882 flags.go:64] FLAG: --cpu-cfs-quota="true"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864076    4882 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864081    4882 flags.go:64] FLAG: --cpu-manager-policy="none"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864086    4882 flags.go:64] FLAG: --cpu-manager-policy-options=""
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864093    4882 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864098    4882 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864103    4882 flags.go:64] FLAG: --enable-debugging-handlers="true"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864108    4882 flags.go:64] FLAG: --enable-load-reader="false"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864113    4882 flags.go:64] FLAG: --enable-server="true"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864117    4882 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864125    4882 flags.go:64] FLAG: --event-burst="100"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864130    4882 flags.go:64] FLAG: --event-qps="50"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864135    4882 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864140    4882 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864145    4882 flags.go:64] FLAG: --eviction-hard=""
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864152    4882 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864156    4882 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864161    4882 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864166    4882 flags.go:64] FLAG: --eviction-soft=""
Oct 30 21:37:56.872469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864170    4882 flags.go:64] FLAG: --eviction-soft-grace-period=""
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864175    4882 flags.go:64] FLAG: --exit-on-lock-contention="false"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864179    4882 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864185    4882 flags.go:64] FLAG: --experimental-mounter-path=""
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864190    4882 flags.go:64] FLAG: --fail-cgroupv1="true"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864195    4882 flags.go:64] FLAG: --fail-swap-on="true"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864199    4882 flags.go:64] FLAG: --feature-gates=""
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864206    4882 flags.go:64] FLAG: --file-check-frequency="20s"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864211    4882 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864216    4882 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864220    4882 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864226    4882 flags.go:64] FLAG: --healthz-port="10248"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864231    4882 flags.go:64] FLAG: --help="false"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864238    4882 flags.go:64] FLAG: --hostname-override="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864243    4882 flags.go:64] FLAG: --housekeeping-interval="10s"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864248    4882 flags.go:64] FLAG: --http-check-frequency="20s"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864253    4882 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864259    4882 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864265    4882 flags.go:64] FLAG: --image-gc-high-threshold="85"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864270    4882 flags.go:64] FLAG: --image-gc-low-threshold="80"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864275    4882 flags.go:64] FLAG: --image-service-endpoint=""
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864280    4882 flags.go:64] FLAG: --kernel-memcg-notification="false"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864284    4882 flags.go:64] FLAG: --kube-api-burst="100"
Oct 30 21:37:56.873262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864289    4882 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864294    4882 flags.go:64] FLAG: --kube-api-qps="50"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864299    4882 flags.go:64] FLAG: --kube-reserved=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864303    4882 flags.go:64] FLAG: --kube-reserved-cgroup=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864308    4882 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864313    4882 flags.go:64] FLAG: --kubelet-cgroups=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864318    4882 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864322    4882 flags.go:64] FLAG: --lock-file=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864327    4882 flags.go:64] FLAG: --log-cadvisor-usage="false"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864331    4882 flags.go:64] FLAG: --log-flush-frequency="5s"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864336    4882 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864345    4882 flags.go:64] FLAG: --log-json-split-stream="false"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864350    4882 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864356    4882 flags.go:64] FLAG: --log-text-split-stream="false"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864361    4882 flags.go:64] FLAG: --logging-format="text"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864366    4882 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864371    4882 flags.go:64] FLAG: --make-iptables-util-chains="true"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864376    4882 flags.go:64] FLAG: --manifest-url=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864380    4882 flags.go:64] FLAG: --manifest-url-header=""
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864387    4882 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864393    4882 flags.go:64] FLAG: --max-open-files="1000000"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864399    4882 flags.go:64] FLAG: --max-pods="110"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864410    4882 flags.go:64] FLAG: --maximum-dead-containers="-1"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864414    4882 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864419    4882 flags.go:64] FLAG: --memory-manager-policy="None"
Oct 30 21:37:56.873934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864423    4882 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864429    4882 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864434    4882 flags.go:64] FLAG: --node-ip="0.0.0.0"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864438    4882 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864449    4882 flags.go:64] FLAG: --node-status-max-images="50"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864454    4882 flags.go:64] FLAG: --node-status-update-frequency="10s"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864459    4882 flags.go:64] FLAG: --oom-score-adj="-999"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864464    4882 flags.go:64] FLAG: --pod-cidr=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864468    4882 flags.go:64] FLAG: --pod-manifest-path=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864473    4882 flags.go:64] FLAG: --pod-max-pids="-1"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864478    4882 flags.go:64] FLAG: --pods-per-core="0"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864483    4882 flags.go:64] FLAG: --port="10250"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864487    4882 flags.go:64] FLAG: --protect-kernel-defaults="false"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864492    4882 flags.go:64] FLAG: --provider-id="aws:///us-west-2a/i-0d7878499bfd6b86e"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864497    4882 flags.go:64] FLAG: --qos-reserved=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864501    4882 flags.go:64] FLAG: --read-only-port="10255"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864506    4882 flags.go:64] FLAG: --register-node="true"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864511    4882 flags.go:64] FLAG: --register-with-taints=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864518    4882 flags.go:64] FLAG: --registry-burst="10"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864523    4882 flags.go:64] FLAG: --registry-qps="5"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864528    4882 flags.go:64] FLAG: --reserved-cpus=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864533    4882 flags.go:64] FLAG: --reserved-memory=""
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864540    4882 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864545    4882 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864549    4882 flags.go:64] FLAG: --rotate-certificates="false"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864554    4882 flags.go:64] FLAG: --rotate-server-certificates="false"
Oct 30 21:37:56.874594 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864559    4882 flags.go:64] FLAG: --runonce="false"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864564    4882 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864569    4882 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864576    4882 flags.go:64] FLAG: --seccomp-default="false"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864580    4882 flags.go:64] FLAG: --serialize-image-pulls="true"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864600    4882 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864606    4882 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864611    4882 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864616    4882 flags.go:64] FLAG: --storage-driver-password="root"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864620    4882 flags.go:64] FLAG: --storage-driver-secure="false"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864625    4882 flags.go:64] FLAG: --storage-driver-table="stats"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864630    4882 flags.go:64] FLAG: --storage-driver-user="root"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864634    4882 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864640    4882 flags.go:64] FLAG: --sync-frequency="1m0s"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864644    4882 flags.go:64] FLAG: --system-cgroups=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864649    4882 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864658    4882 flags.go:64] FLAG: --system-reserved-cgroup=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864663    4882 flags.go:64] FLAG: --tls-cert-file=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864668    4882 flags.go:64] FLAG: --tls-cipher-suites="[]"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864674    4882 flags.go:64] FLAG: --tls-min-version=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864679    4882 flags.go:64] FLAG: --tls-private-key-file=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864683    4882 flags.go:64] FLAG: --topology-manager-policy="none"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864688    4882 flags.go:64] FLAG: --topology-manager-policy-options=""
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864693    4882 flags.go:64] FLAG: --topology-manager-scope="container"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864700    4882 flags.go:64] FLAG: --v="2"
Oct 30 21:37:56.875272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864707    4882 flags.go:64] FLAG: --version="false"
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864714    4882 flags.go:64] FLAG: --vmodule=""
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864720    4882 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.864734    4882 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.864990    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.864997    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865001    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865006    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865010    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865015    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865022    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865026    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865030    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865034    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865039    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865043    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865047    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865051    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865055    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865060    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.876073 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865064    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865068    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865072    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865077    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865082    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865086    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865090    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865094    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865099    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865103    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865107    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865113    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865118    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865122    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865126    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865130    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865135    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865139    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865144    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.876674 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865148    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865152    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865156    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865161    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865166    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865170    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865175    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865179    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865183    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865188    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865193    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865197    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865201    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865205    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865209    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865213    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865217    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865221    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865225    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.877133 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865229    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865233    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865237    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865241    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865245    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865251    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865258    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865263    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865268    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865273    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865278    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865283    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865290    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865296    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865301    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865305    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865313    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865319    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.877627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865326    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865332    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865339    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865343    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865348    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865353    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865357    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865361    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865365    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865370    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865374    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865379    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865383    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865387    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865414    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865421    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865426    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865431    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865435    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865441    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.878102 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865446    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865450    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865454    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865458    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865462    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865467    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865472    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865476    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865480    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865484    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865490    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865495    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865499    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865503    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.865544    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865617    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.878681 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865626    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865632    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865636    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865641    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865647    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865651    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865655    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865660    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865664    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865669    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865673    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865679    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865686    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865691    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865695    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865699    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865708    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865713    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865718    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.879099 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865723    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865727    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865732    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865736    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865741    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865745    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865750    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865756    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865760    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865765    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865769    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865773    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865778    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865789    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865793    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865797    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865803    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865807    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865812    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.879667 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865816    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865837    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865841    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865846    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865850    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865854    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865859    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865863    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865868    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865872    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865878    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865884    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865888    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865892    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865897    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865901    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865905    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865910    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865915    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865919    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.880159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865925    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865930    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865934    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865938    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865942    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865947    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865952    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865956    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865960    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865964    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865968    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865971    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865976    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865981    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865986    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865990    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865994    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.865999    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866003    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.880660 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866008    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866012    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866017    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866024    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866028    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866033    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866037    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866041    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866046    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866050    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866054    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866059    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866064    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866070    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866077    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866081    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866086    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866090    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866095    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866100    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.881111 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866104    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866108    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866112    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866117    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866121    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866125    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866130    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.866134    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870040    4882 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870057    4882 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870074    4882 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870079    4882 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870136    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870141    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870146    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870149    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870152    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870155    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870158    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870161    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.881582 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870163    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870166    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870169    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870172    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870174    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870177    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870180    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870182    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870185    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870188    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870190    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870193    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870196    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870198    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870201    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870203    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870206    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870209    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870212    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.882154 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870215    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870218    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870221    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870224    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870227    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870229    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870232    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870234    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870237    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870240    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870242    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870245    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870247    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870250    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870252    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870255    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870258    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870260    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870263    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.882633 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870265    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870268    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870271    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870273    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870277    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870281    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870284    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870287    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870290    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870293    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870296    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870298    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870301    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870304    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870307    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870309    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870312    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870315    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870318    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870320    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.883117 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870322    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870325    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870328    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870330    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870333    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870335    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870338    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870340    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870343    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870345    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870348    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870351    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870353    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870357    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870360    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870363    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870366    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870368    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870371    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.883757 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870374    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870376    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870379    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870382    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870384    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870387    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870390    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870393    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870396    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870398    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870401    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870404    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870406    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870410    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870414    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870417    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870420    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870422    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870425    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.884213 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870427    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870430    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870457    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870611    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870616    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870619    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870622    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870625    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870627    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870630    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870633    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870636    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870638    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870641    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870644    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870647    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.884690 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870650    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870654    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870659    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870662    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870664    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870667    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870670    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870672    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870675    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870678    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870681    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870683    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870686    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870688    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870691    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870694    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870697    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870699    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870702    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.885071 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870704    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870707    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870709    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870712    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870714    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870717    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870720    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870722    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870725    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870728    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870730    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870733    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870736    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870739    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870742    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870745    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870748    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870750    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870753    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870755    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.885516 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870758    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870760    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870764    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870766    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870769    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870771    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870773    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870776    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870779    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870781    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870784    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870786    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870789    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870791    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870794    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870796    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870799    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870801    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870804    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.886040 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870807    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870810    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870812    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870815    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870817    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870820    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870822    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870825    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870827    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870830    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870833    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870835    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870838    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870840    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870844    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870848    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870851    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870854    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870857    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.886508 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870860    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870862    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870864    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870867    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870869    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870872    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870874    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870877    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870879    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870882    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870885    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870887    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870890    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870892    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870895    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870897    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.886980 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.870922    4882 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870953    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870956    4882 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870960    4882 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870963    4882 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870967    4882 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870971    4882 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870974    4882 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870976    4882 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870979    4882 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870982    4882 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870984    4882 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870987    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870989    4882 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870992    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870995    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.870998    4882 feature_gate.go:386] unrecognized feature gate: EVPN
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871000    4882 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871002    4882 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Oct 30 21:37:56.887396 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871005    4882 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871008    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871010    4882 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871012    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871087    4882 feature_gate.go:386] unrecognized feature gate: DualReplica
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871093    4882 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871096    4882 feature_gate.go:386] unrecognized feature gate: Example2
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871100    4882 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871105    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871109    4882 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871514    4882 feature_gate.go:386] unrecognized feature gate: SignatureStores
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871527    4882 feature_gate.go:386] unrecognized feature gate: OVNObservability
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871535    4882 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871543    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871551    4882 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871559    4882 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871577    4882 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871607    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871618    4882 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871625    4882 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Oct 30 21:37:56.887936 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871633    4882 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871642    4882 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871653    4882 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871657    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871662    4882 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871667    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871674    4882 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871679    4882 feature_gate.go:386] unrecognized feature gate: OSStreams
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871686    4882 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871697    4882 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871701    4882 feature_gate.go:386] unrecognized feature gate: EventTTL
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871707    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871713    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871719    4882 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871724    4882 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871730    4882 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871734    4882 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871738    4882 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871744    4882 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Oct 30 21:37:56.888427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871747    4882 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871751    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871761    4882 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871770    4882 feature_gate.go:386] unrecognized feature gate: NewOLM
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871775    4882 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871780    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871784    4882 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871788    4882 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871795    4882 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871799    4882 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871804    4882 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871808    4882 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871813    4882 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871818    4882 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871828    4882 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871833    4882 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871839    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871843    4882 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871849    4882 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871854    4882 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Oct 30 21:37:56.888955 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871858    4882 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871862    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871867    4882 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871872    4882 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871876    4882 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871880    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871884    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871893    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871898    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871902    4882 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871906    4882 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871910    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871914    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871919    4882 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871924    4882 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871930    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871935    4882 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871941    4882 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871948    4882 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Oct 30 21:37:56.889675 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871957    4882 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871961    4882 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871965    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871970    4882 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871974    4882 feature_gate.go:386] unrecognized feature gate: Example
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871978    4882 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871983    4882 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871987    4882 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871992    4882 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:56.871996    4882 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.872113    4882 server.go:957] "Client rotation is on, will bootstrap in background"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.873498    4882 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.873565    4882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.873789    4882 server.go:1014] "Starting client certificate rotation"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.873936    4882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.873991    4882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 21:31:45 +0000 UTC" deadline="2028-03-30 16:20:52.872455934 +0000 UTC"
Oct 30 21:37:56.890473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.874037    4882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12402h42m55.998420635s"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.874276    4882 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.875431    4882 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.876713    4882 log.go:25] "Validated CRI v1 runtime API"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.878377    4882 log.go:25] "Validated CRI v1 image API"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.879267    4882 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Oct 30 21:37:56.891114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.881029    4882 fs.go:135] Filesystem UUIDs: map[166dc5c4-397d-4d6a-844f-e3013f9894e7:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 f61f6cc5-e4e7-4428-a1ea-fff8a5a72e0c:/dev/nvme0n1p3]
Oct 30 21:37:56.891374 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.881050    4882 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2/userdata/shm major:0 minor:248 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b/userdata/shm major:0 minor:225 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328/userdata/shm major:0 minor:81 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef/userdata/shm major:0 minor:98 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90/userdata/shm major:0 minor:270 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d/userdata/shm major:0 minor:268 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d/userdata/shm major:0 minor:219 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c/userdata/shm major:0 minor:49 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa/userdata/shm major:0 minor:214 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3/userdata/shm major:0 minor:266 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/07f98edf-bb0a-422e-ab8f-c1f935774f2a/volumes/kubernetes.io~projected/kube-api-access-fvw5h:{mountpoint:/var/lib/kubelet/pods/07f98edf-bb0a-422e-ab8f-c1f935774f2a/volumes/kubernetes.io~projected/kube-api-access-fvw5h major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749/volumes/kubernetes.io~projected/kube-api-access-6jw5z:{mountpoint:/var/lib/kubelet/pods/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749/volumes/kubernetes.io~projected/kube-api-access-6jw5z major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2328abe6-8360-4e86-9a59-4e308704aac1/volumes/kubernetes.io~projected/kube-api-access-mfsf7:{mountpoint:/var/lib/kubelet/pods/2328abe6-8360-4e86-9a59-4e308704aac1/volumes/kubernetes.io~projected/kube-api-access-mfsf7 major:0 minor:218 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/2bf2774c-7d35-4bff-98ae-489cc598b066/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/2bf2774c-7d35-4bff-98ae-489cc598b066/volumes/kubernetes.io~secret/agent-certs major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3b4f2e52-d72d-4962-9627-e1817b1774a9/volumes/kubernetes.io~projected/kube-api-access-vgn88:{mountpoint:/var/lib/kubelet/pods/3b4f2e52-d72d-4962-9627-e1817b1774a9/volumes/kubernetes.io~projected/kube-api-access-vgn88 major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79/volumes/kubernetes.io~projected/kube-api-access-65tq4:{mountpoint:/var/lib/kubelet/pods/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79/volumes/kubernetes.io~projected/kube-api-access-65tq4 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~projected/kube-api-access-h5jns:{mountpoint:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~projected/kube-api-access-h5jns major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/79565b90-0624-4317-983b-6f44b73c6f78/volumes/kubernetes.io~projected/kube-api-access-67xtz:{mountpoint:/var/lib/kubelet/pods/79565b90-0624-4317-983b-6f44b73c6f78/volumes/kubernetes.io~projected/kube-api-access-67xtz major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8ee90e5a-7c9f-4b24-841c-86b50175efd6/volumes/kubernetes.io~projected/kube-api-access-tq8jv:{mountpoint:/var/lib/kubelet/pods/8ee90e5a-7c9f-4b24-841c-86b50175efd6/volumes/kubernetes.io~projected/kube-api-access-tq8jv major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~projected/kube-api-access-h7pqv:{mountpoint:/var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~projected/kube-api-access-h7pqv major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abb604c9-145e-4bdb-8679-a5a0c25b1441/volumes/kubernetes.io~projected/kube-api-access-vvgsw:{mountpoint:/var/lib/kubelet/pods/abb604c9-145e-4bdb-8679-a5a0c25b1441/volumes/kubernetes.io~projected/kube-api-access-vvgsw major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~projected/kube-api-access-fqjqw:{mountpoint:/var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~projected/kube-api-access-fqjqw major:0 minor:224 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~secret/signing-key:{mountpoint:/var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~secret/signing-key major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d97ac33e-24e8-4256-a3cd-154231e10553/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/d97ac33e-24e8-4256-a3cd-154231e10553/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/tmp major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~projected/kube-api-access-btqkw:{mountpoint:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~projected/kube-api-access-btqkw major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~projected/kube-api-access-ggqtk:{mountpoint:/var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~projected/kube-api-access-ggqtk major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~secret/metrics-tls major:0 minor:243 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f0d8c917-e87e-4304-bec9-66f213536b88/volumes/kubernetes.io~projected/kube-api-access-9k26r:{mountpoint:/var/lib/kubelet/pods/f0d8c917-e87e-4304-bec9-66f213536b88/volumes/kubernetes.io~projected/kube-api-access-9k26r major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~projected/kube-api-access-djmm6:{mountpoint:/var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~projected/kube-api-access-djmm6 major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~secret/metrics-certs major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f3de0e99-5c49-4347-878a-58dbd3fd404b/volumes/kubernetes.io~projected/kube-api-access-zmrfq:{mountpoint:/var/lib/kubelet/pods/f3de0e99-5c49-4347-878a-58dbd3fd404b/volumes/kubernetes.io~projected/kube-api-access-zmrfq major:0 minor:213 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/948e5642df96befa228e8f8db81be7e932c2974b62f367450ed86764fb1c9023/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d0d1f4c3f920f130dc78c3fa26957dea6fd7c8db08fa62018f0992febadafc4f/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/51eed127ffaabef7ab32f5b3cb0d8faed2d9d90b61deefdc3b784a03ac499637/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-108:{mountpoint:/var/lib/containers/storage/overlay/00e5bba7093b050d28155f97967213b34055cd45c84696ec44cc78777913152f/merged major:0 minor:108 fsType:overlay blockSize:0} overlay_0-111:{mountpoint:/var/lib/containers/storage/overlay/f800e7e407a648791e67a0f1a47d937efbee502b6717ac52ec5cfd103635156b/merged major:0 minor:111 fsType:overlay blockSize:0} overlay_0-116:{mountpoint:/var/lib/containers/storage/overlay/28958177629ec132df64a48874caa3361928565bf339f3b7e844c6d06bbd791c/merged major:0 minor:116 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/3960e723a580229dcb1a7e3adf7f14e2e437fbdc8697b616de83b171a143fa43/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/c5d71fa4db3f7d3323886584976e5a4c64635dc6a5eac4bb07301a451520245e/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/42c67f9d0901755c77cfc40cb94fed0a14bd2daa652261c46a155a1987ede762/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/ad92abc78d54fb43cf0e5de95f92c88198561e701933d89430bec7fa7f359a65/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/d2f62a17ad947892b03dae33ed40567cfc6fbc9066793ca3d93887eba26bc5a5/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/b67e49b5d0a06ebe0eccaa321cf3453a5b32cfb37595cbfa3e6fe7fb372cc37c/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/a527ebb34216ced1d33afde243d4cfb839822c67e42d314f72fbb1906543c335/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/b13a36ddf3f86ca5013ffa957fe560e8e02fe33f7e0ef88cc4bbcd1d22daf39e/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/3e9d629d80466fae2f76688e5bc638c54d333d316b318f2225459f3c7b77fcbb/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/9a151086f678b85dd6febf8b285099f54189d358f5e6839e2a8528a85e9f9b28/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/9426cf3a1970f77e1339f0871ce3c9804896678f66933463d85d0836c848c189/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/99110c55f5aee9a732ff091445e5896d3b4ebf42f42269ab81cad2482d4bd186/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/6584975c261102dab15f7fcaeba70e5760a6850fb50796d1bdfcc0a9afb0e856/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/885108b4a84d15785aede04ef53165e2529194efad253fab03bb20f781ed30b9/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/da7b80e87b8ea9da8cb6ba1a8ba81b4678121bbface5b8b044055086b1ddcfad/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/687fc246f49919002cfe4c8755ce13df7d7c7d79866adbf6e5bae4dd7c29f79f/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/81d379676623b8be177e195437e23bce55acdd5c009f91eea4a738543fceabe3/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-216:{mountpoint:/var/lib/containers/storage/overlay/2858a7c3601ed6d25c7eecf2c81c41783ee2e03253bd185458c0c92081fb0fc6/merged major:0 minor:216 fsType:overlay blockSize:0} overlay_0-222:{mountpoint:/var/lib/containers/storage/overlay/7af5f73cc8e5725cbd3cdf2286d3119f60ed715a803e08bcc0002fd2095a3f6e/merged major:0 minor:222 fsType:overlay blockSize:0} overlay_0-227:{mountpoint:/var/lib/containers/storage/overlay/f7779cb2e8964e3d6d201b0019421e6c09d43d17c1ff721a2b4af09464dac24b/merged major:0 minor:227 fsType:overlay blockSize:0} overlay_0-229:{mountpoint:/var/lib/containers/storage/overlay/1064de3b0cad4b0503dfb34c7c9a485ec8199e40de6a6363e9288bfc39b82ca1/merged major:0 minor:229 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/639ebe11c859fc84ef81d0c693b340f8a076440973887c49ed6421adc9931a0e/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-237:{mountpoint:/var/lib/containers/storage/overlay/4c2d5425d04a76dc1900e29b473c88aeb092b10a40a6a99f45f9f882a3cef81e/merged major:0 minor:237 fsType:overlay blockSize:0} overlay_0-250:{mountpoint:/var/lib/containers/storage/overlay/46ac619eb936771b9e6e66a54396da08b27ac090ca6c4d813a79a49e5ad264b9/merged major:0 minor:250 fsType:overlay blockSize:0} overlay_0-252:{mountpoint:/var/lib/containers/storage/overlay/41b0c037b34313fb797ca83824baccffd1e2ddd45a1ae1f872c6355a40d57363/merged major:0 minor:252 fsType:overlay blockSize:0} overlay_0-254:{mountpoint:/var/lib/containers/storage/overlay/f9120d89db1fd70d8414b46190d63ca1df91a995054dec8342e37cf22634266c/merged major:0 minor:254 fsType:overlay blockSize:0} overlay_0-272:{mountpoint:/var/lib/containers/storage/overlay/6de32d83b0e4cd684ffeee3f145b22a8ce2af72acef55b669494ac595f875d8d/merged major:0 minor:272 fsType:overlay blockSize:0} overlay_0-274:{mountpoint:/var/lib/containers/storage/overlay/49ff8418e6f7e66666ba792dfb506ca35e2dfc9f5fb14079b21392934b43ca94/merged major:0 minor:274 fsType:overlay blockSize:0} overlay_0-276:{mountpoint:/var/lib/containers/storage/overlay/3ada2925f703d26c54f20dbba34c4ff3395620d831281965eeb6bf85bd2ccef0/merged major:0 minor:276 fsType:overlay blockSize:0} overlay_0-278:{mountpoint:/var/lib/containers/storage/overlay/488b46997e323e43f45948e3a2a40b74cfb9fe598755bcb75ebbf2cd69bffd42/merged major:0 minor:278 fsType:overlay blockSize:0} overlay_0-280:{mountpoint:/var/lib/containers/storage/overlay/17bf08373e16f40d9951e0924eaaaff8a0df4600684df93f517b81304acb46d0/merged major:0 minor:280 fsType:overlay blockSize:0} overlay_0-286:{mountpoint:/var/lib/containers/storage/overlay/41a5d066ae105389d4fe8cc9eb5162176fa9eef89b0c3df812cb2e49ff1ad5df/merged major:0 minor:286 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/21d40b48f7b9c7dc49b44d268fb3999fdea8d05f69d9597f1ae01a6df85b1e8c/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/6dd5e6ea5b01fcced3aff290912701f83ebd130a888ecf6f468e2917abd6ca49/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/d5514467b8f322cdef12a2f617c87a975be1fcb88a555ec19ef58be9d6171ed5/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/8fe38a4a59b9b88ffa4b329622e18315ff90ccd1efa3338a82eb29fabea1dee1/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/f4e65243dcd4c48d17873158282b07a339aadc515ba84600a3d7d6bcacf7940e/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/d2fd52af8d2aa1b8e7e87600f9fbecabf96c5aa92cbcd7db05e37bf41c2541dc/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/0fa0f53e72e2ec7040e0d37b44afc4de767754f6c05b1bb57090164ff7625a16/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/c0722068178146c5d6feef73a16fe65cbce1afc93696a585f71762b0dfea23ef/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-89:{mountpoint:/var/lib/containers/storage/overlay/08baf8411d7d0b9e5ee0e9158611938fc25fb1dadb3671ad7daad3741defb625/merged major:0 minor:89 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/6094289761b987107f8faaf0ca878eac89d2308a5a43da3e7b9de2955f0f5e7b/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/ebfb555333cbbf4dada9efa4b1303ed00dc9bd10cf5ba4d93fe9c14c2767444f/merged major:0 minor:96 fsType:overlay blockSize:0}]
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.887561    4882 manager.go:217] Machine: {Timestamp:2026-10-30 21:37:56.887013563 +0000 UTC m=+0.060337981 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201671 MemoryCapacity:33164324864 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec22f9a72f1e34050867a8cf0fd7ebb7 SystemUUID:ec22f9a7-2f1e-3405-0867-a8cf0fd7ebb7 BootID:835dcc53-c7fb-43b1-b3b3-f27d56f3a9b3 Filesystems:[{Device:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~projected/kube-api-access-h5jns DeviceMajor:0 DeviceMinor:67 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/61ac3336bd7338a1d40d8332bd463f43715af6aab238226f6498028962f437ef/userdata/shm DeviceMajor:0 DeviceMinor:98 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f3de0e99-5c49-4347-878a-58dbd3fd404b/volumes/kubernetes.io~projected/kube-api-access-zmrfq DeviceMajor:0 DeviceMinor:213 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-278 DeviceMajor:0 DeviceMinor:278 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749/volumes/kubernetes.io~projected/kube-api-access-6jw5z DeviceMajor:0 DeviceMinor:57 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-108 DeviceMajor:0 DeviceMinor:108 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~secret/signing-key DeviceMajor:0 DeviceMinor:221 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/b5da833b-0939-4f20-a2b3-18fe731a9360/volumes/kubernetes.io~projected/kube-api-access-fqjqw DeviceMajor:0 DeviceMinor:224 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-280 DeviceMajor:0 DeviceMinor:280 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-111 DeviceMajor:0 DeviceMinor:111 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ab7b5825ee5862dd0709db4a71e5716a83c0aedba338d3399b861c2b2adcb5b/userdata/shm DeviceMajor:0 DeviceMinor:225 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~projected/kube-api-access-djmm6 DeviceMajor:0 DeviceMinor:56 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/79565b90-0624-4317-983b-6f44b73c6f78/volumes/kubernetes.io~projected/kube-api-access-67xtz DeviceMajor:0 DeviceMinor:58 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/abb604c9-145e-4bdb-8679-a5a0c25b1441/volumes/kubernetes.io~projected/kube-api-access-vvgsw DeviceMajor:0 DeviceMinor:208 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-250 DeviceMajor:0 DeviceMinor:250 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-254 DeviceMajor:0 DeviceMinor:254 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-276 DeviceMajor:0 DeviceMinor:276 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79/volumes/kubernetes.io~projected/kube-api-access-65tq4 DeviceMajor:0 DeviceMinor:64 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/07f98edf-bb0a-422e-ab8f-c1f935774f2a/volumes/kubernetes.io~projected/kube-api-access-fvw5h DeviceMajor:0 DeviceMinor:55 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16582164480 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3ce4fdd7e08cae308404c85c9ea4d0db988956411e1f960fadb6e4f3abfce2dc/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-216 DeviceMajor:0 DeviceMinor:216 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:62 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4e6b3f54c6ee45296f980af76bfa45a8a5bc267e6a907c9ec211869d1e0c8328/userdata/shm DeviceMajor:0 DeviceMinor:81 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/71cea7ef8c9b1cba364c1c9c50ecae29f469399a2063ca6868d5828356bcdc5b/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a8b683cc8f9eaba434a1e744b1d9ea9dd1272e206e125302ad0ecc8f2eb404eb/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-222 DeviceMajor:0 DeviceMinor:222 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:243 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/3b4f2e52-d72d-4962-9627-e1817b1774a9/volumes/kubernetes.io~projected/kube-api-access-vgn88 DeviceMajor:0 DeviceMinor:265 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/2bf2774c-7d35-4bff-98ae-489cc598b066/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:63 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-227 DeviceMajor:0 DeviceMinor:227 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-237 DeviceMajor:0 DeviceMinor:237 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6632865792 Type:vfs Inodes:819200 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c/userdata/shm DeviceMajor:0 DeviceMinor:49 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0f3e95fea3d5436f91ff7cfaf900e0a33ae6a2a5fafaed577c6791baef017018/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa/userdata/shm DeviceMajor:0 DeviceMinor:214 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:60 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~projected/kube-api-access-btqkw DeviceMajor:0 DeviceMinor:66 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-116 DeviceMajor:0 DeviceMinor:116 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ef440513-6ad3-4663-92fb-1b69c378c66f/volumes/kubernetes.io~projected/kube-api-access-ggqtk DeviceMajor:0 DeviceMinor:197 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/1b57304fc28760edc5efbe07713110b678d17989c0447ddb8181f45a8914cca2/userdata/shm DeviceMajor:0 DeviceMinor:248 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16582160384 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7/volumes/kubernetes.io~projected/kube-api-access-h7pqv DeviceMajor:0 DeviceMinor:201 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/797ca8240c8c9ceb05099cb399ddb2d7c93135c360c928adad36e19f8ed49f8d/userdata/shm DeviceMajor:0 DeviceMinor:268 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/8ee90e5a-7c9f-4b24-841c-86b50175efd6/volumes/kubernetes.io~projected/kube-api-access-tq8jv DeviceMajor:0 DeviceMinor:59 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e6be0ce9b096a0ad33c90860e134efe7cdab1617b79dbdde6b4cab89f19cb2f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/2328abe6-8360-4e86-9a59-4e308704aac1/volumes/kubernetes.io~projected/kube-api-access-mfsf7 DeviceMajor:0 DeviceMinor:218 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-229 DeviceMajor:0 DeviceMinor:229 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eb8188824ba0b60135879e2b534ef3c5f5db2504845024374ab72b423332a4e3/userdata/shm DeviceMajor:0 DeviceMinor:266 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e021af5f-e7c2-474e-aff2-7cc532fbcaf0/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:61 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-252 DeviceMajor:0 DeviceMinor:252 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-286 DeviceMajor:0 DeviceMinor:286 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b4cd7936cc9f727984fcedb83e844b02d441803ac115744d114d70ade07b5fba/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bd4556663ada2a31e7c1443aea5d889ba4125f012b392d2410c4e2abd8bd510d/userdata/shm DeviceMajor:0 DeviceMinor:219 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/d97ac33e-24e8-4256-a3cd-154231e10553/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:264 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/var/lib/kubelet/pods/f0d8c917-e87e-4304-bec9-66f213536b88/volumes/kubernetes.io~projected/kube-api-access-9k26r DeviceMajor:0 DeviceMinor:65 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-89 DeviceMajor:0 DeviceMinor:89 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f1d1b936-7975-413e-8b25-ace2bd71cdd3/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:260 Capacity:31985725440 Type:vfs Inodes:4048379 HasInodes:true} {Device:/run/containers/storage/overlay-containers/72be0e4bc3d1f9b5055ac6ae24cc99431d5740152a922c36c227a6d480de4b90/userdata/shm DeviceMajor:0 DeviceMinor:270 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true} {Device:overlay_0-272 DeviceMajor:0 DeviceMinor:272 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-274 DeviceMajor:0 DeviceMinor:274 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/45a222cdf773231e42162d6edb1153274813e781c5335606409d37c1ed5358c3/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4048379 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:1b57304fc28760e MacAddress:3a:30:0b:63:73:ad Speed:10000 Mtu:8901} {Name:3ab7b5825ee5862 MacAddress:2e:58:5b:2d:52:fd Speed:10000 Mtu:8901} {Name:72be0e4bc3d1f9b MacAddress:4a:45:24:f5:6a:77 Speed:10000 Mtu:8901} {Name:797ca8240c8c9ce MacAddress:ee:4e:f8:c3:83:9b Speed:10000 Mtu:8901} {Name:a8b683cc8f9eaba MacAddress:8a:65:f5:a3:36:70 Speed:10000 Mtu:8901} {Name:bd4556663ada2a3 MacAddress:56:12:a5:01:96:fb Speed:10000 Mtu:8901} {Name:br-ex MacAddress:06:ff:cc:e2:39:7d Speed:0 Mtu:9001} {Name:br-int MacAddress:7a:fe:be:6d:61:6f Speed:0 Mtu:8901} {Name:ce42b28ba6f668d MacAddress:5e:4f:56:43:46:5a Speed:10000 Mtu:8901} {Name:eb8188824ba0b60 MacAddress:1e:f6:c9:2e:19:00 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:06:ff:cc:e2:39:7d Speed:0 Mtu:9001} {Name:genev_sys_6081 MacAddress:86:9f:20:57:7f:34 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:84:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:32:58:4d:46:c9:16 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164324864 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.887849    4882 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.887897    4882 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888238    4882 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888261    4882 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-82.us-west-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888395    4882 topology_manager.go:143] "Creating topology manager with none policy"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888403    4882 container_manager_linux.go:311] "Creating device plugin manager"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888414    4882 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888427    4882 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888531    4882 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888731    4882 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888847    4882 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888884    4882 kubelet.go:493] "Attempting to sync node with API server"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888896    4882 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888907    4882 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888916    4882 kubelet.go:405] "Adding apiserver pod source"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.888926    4882 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.889856    4882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.889874    4882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:37:56.891646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.890742    4882 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Oct 30 21:37:56.895044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895007    4882 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Oct 30 21:37:56.895044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895044    4882 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Oct 30 21:37:56.895190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895177    4882 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Oct 30 21:37:56.895473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895458    4882 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Oct 30 21:37:56.895515 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895505    4882 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Oct 30 21:37:56.895660 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895647    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Oct 30 21:37:56.895705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895667    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Oct 30 21:37:56.895705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895676    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Oct 30 21:37:56.895705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895686    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Oct 30 21:37:56.895705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895695    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Oct 30 21:37:56.895705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895703    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895712    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895720    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895731    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895739    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895753    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895766    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895800    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Oct 30 21:37:56.895866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.895809    4882 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Oct 30 21:37:56.899269 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.899255    4882 server.go:1263] "Started kubelet"
Oct 30 21:37:56.901140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.899314    4882 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Oct 30 21:37:56.901140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.899360    4882 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Oct 30 21:37:56.901140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.899632    4882 server_v1.go:49] "podresources" method="list" useActivePods=true
Oct 30 21:37:56.901140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.899968    4882 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Oct 30 21:37:56.901140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.900431    4882 server.go:317] "Adding debug handlers to kubelet server"
Oct 30 21:37:56.899835 ip-10-0-141-82 systemd[1]: Started Kubernetes Kubelet.
Oct 30 21:37:56.902368 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902349    4882 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Oct 30 21:37:56.902462 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902418    4882 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:56.902526    4882 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902539    4882 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 21:31:46 +0000 UTC" deadline="2028-05-18 04:56:41.452951823 +0000 UTC"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902608    4882 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13567h18m44.55037302s"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902616    4882 volume_manager.go:309] "The desired_state_of_world populator starts"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902627    4882 volume_manager.go:311] "Starting Kubelet Volume Manager"
Oct 30 21:37:56.903210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.902724    4882 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Oct 30 21:37:56.904292 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904275    4882 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Oct 30 21:37:56.904292 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904291    4882 factory.go:55] Registering systemd factory
Oct 30 21:37:56.904429 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904300    4882 factory.go:223] Registration of the systemd container factory successfully
Oct 30 21:37:56.904659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904640    4882 factory.go:153] Registering CRI-O factory
Oct 30 21:37:56.904760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904662    4882 factory.go:223] Registration of the crio container factory successfully
Oct 30 21:37:56.904760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904682    4882 factory.go:103] Registering Raw factory
Oct 30 21:37:56.904760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904695    4882 manager.go:1199] Started watching for new ooms in manager
Oct 30 21:37:56.904898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904839    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1a38f40-6f73-4d05-9ca4-d82518e4bbe7" volumeName="kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert" seLinuxMountContext=""
Oct 30 21:37:56.904898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904877    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb604c9-145e-4bdb-8679-a5a0c25b1441" volumeName="kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume" seLinuxMountContext=""
Oct 30 21:37:56.904898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904892    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef440513-6ad3-4663-92fb-1b69c378c66f" volumeName="kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904906    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3de0e99-5c49-4347-878a-58dbd3fd404b" volumeName="kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904920    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f98edf-bb0a-422e-ab8f-c1f935774f2a" volumeName="kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904935    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa" volumeName="kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904948    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d97ac33e-24e8-4256-a3cd-154231e10553" volumeName="kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904962    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef440513-6ad3-4663-92fb-1b69c378c66f" volumeName="kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904979    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f98edf-bb0a-422e-ab8f-c1f935774f2a" volumeName="kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.904992    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3b4f2e52-d72d-4962-9627-e1817b1774a9" volumeName="kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905004    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="79565b90-0624-4317-983b-6f44b73c6f78" volumeName="kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905017    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb604c9-145e-4bdb-8679-a5a0c25b1441" volumeName="kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw" seLinuxMountContext=""
Oct 30 21:37:56.905038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905030    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef440513-6ad3-4663-92fb-1b69c378c66f" volumeName="kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905049    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3de0e99-5c49-4347-878a-58dbd3fd404b" volumeName="kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905065    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2328abe6-8360-4e86-9a59-4e308704aac1" volumeName="kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905077    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bf2774c-7d35-4bff-98ae-489cc598b066" volumeName="kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905090    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="2bf2774c-7d35-4bff-98ae-489cc598b066" volumeName="kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905097    4882 manager.go:319] Starting recovery of all containers
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905101    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa" volumeName="kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905115    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d1b936-7975-413e-8b25-ace2bd71cdd3" volumeName="kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905126    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3de0e99-5c49-4347-878a-58dbd3fd404b" volumeName="kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905142    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5da833b-0939-4f20-a2b3-18fe731a9360" volumeName="kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905154    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e021af5f-e7c2-474e-aff2-7cc532fbcaf0" volumeName="kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905166    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ef440513-6ad3-4663-92fb-1b69c378c66f" volumeName="kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905177    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f1d1b936-7975-413e-8b25-ace2bd71cdd3" volumeName="kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6" seLinuxMountContext=""
Oct 30 21:37:56.905496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905191    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f3de0e99-5c49-4347-878a-58dbd3fd404b" volumeName="kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905203    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f98edf-bb0a-422e-ab8f-c1f935774f2a" volumeName="kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905540    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="07f98edf-bb0a-422e-ab8f-c1f935774f2a" volumeName="kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905625    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6a00c1-5c3d-48e9-974d-9ac8477c2f79" volumeName="kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905660    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a1a38f40-6f73-4d05-9ca4-d82518e4bbe7" volumeName="kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905680    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5da833b-0939-4f20-a2b3-18fe731a9360" volumeName="kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905708    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e021af5f-e7c2-474e-aff2-7cc532fbcaf0" volumeName="kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905722    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0d8c917-e87e-4304-bec9-66f213536b88" volumeName="kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905741    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e474a6c-d260-4e3e-bdb9-d3a00d5e3749" volumeName="kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905754    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e474a6c-d260-4e3e-bdb9-d3a00d5e3749" volumeName="kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905773    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa" volumeName="kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905799    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa" volumeName="kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905815    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e021af5f-e7c2-474e-aff2-7cc532fbcaf0" volumeName="kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905852    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f0d8c917-e87e-4304-bec9-66f213536b88" volumeName="kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905866    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d6a00c1-5c3d-48e9-974d-9ac8477c2f79" volumeName="kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905884    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa" volumeName="kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905899    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ee90e5a-7c9f-4b24-841c-86b50175efd6" volumeName="kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905915    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abb604c9-145e-4bdb-8679-a5a0c25b1441" volumeName="kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905935    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b5da833b-0939-4f20-a2b3-18fe731a9360" volumeName="kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905949    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="1e474a6c-d260-4e3e-bdb9-d3a00d5e3749" volumeName="kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.905968    4882 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8ee90e5a-7c9f-4b24-841c-86b50175efd6" volumeName="kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv" seLinuxMountContext=""
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.906024    4882 reconstruct.go:97] "Volume reconstruction finished"
Oct 30 21:37:56.906116 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.906033    4882 reconciler.go:29] "Reconciler: start to sync state"
Oct 30 21:37:56.933705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.933644    4882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Oct 30 21:37:56.934959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.934933    4882 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Oct 30 21:37:56.934959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.934959    4882 status_manager.go:249] "Starting to sync pod status with apiserver"
Oct 30 21:37:56.935123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.934981    4882 kubelet.go:2521] "Starting kubelet main sync loop"
Oct 30 21:37:56.935123 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:56.935025    4882 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Oct 30 21:37:56.949026 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.948995    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="3d14f8003915a8917e146b42d6f7a7a5e06285f845b9603567aa39518f0e23aa" exitCode=0
Oct 30 21:37:56.949026 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.949021    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="17b3886ad1259fabadd023c7f6a524ed08e2832b787090dec0902ceb0355394f" exitCode=0
Oct 30 21:37:56.949026 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.949029    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="4e3c1cae641e35e914277aaa560a5d504948557f13798cd58869793a1e21b0fe" exitCode=0
Oct 30 21:37:56.949026 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.949035    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="c581dd26c2fd543e61e1bd25b479a44a2af40106628a620953c6b2ca37cab3e5" exitCode=0
Oct 30 21:37:56.949280 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.949040    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="73476c3b26aeb4f738d57aad80ab944a5e00be5e522c756153ce8d99c1095b1e" exitCode=0
Oct 30 21:37:56.949280 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.949045    4882 generic.go:356] "Generic (PLEG): container finished" podID="07f98edf-bb0a-422e-ab8f-c1f935774f2a" containerID="158a926b33cc7e282cfa6b749be03fd5902196af85b89c408d5906d631bbe6a5" exitCode=0
Oct 30 21:37:56.970002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.969967    4882 generic.go:356] "Generic (PLEG): container finished" podID="6b11aa0892d28c2064080199cdadd8ae" containerID="d7af3f5fc7a5c7e23cca5703413244b70b22aa32f6cdbeeaea0f9ddfafc2ac1d" exitCode=0
Oct 30 21:37:56.982483 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.982456    4882 manager.go:324] Recovery completed
Oct 30 21:37:56.998519 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:56.998495    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.001366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.001345    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.001433 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.001379    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.001433 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.001389    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.002967 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.002935    4882 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:37:57.003432 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003417    4882 cpu_manager.go:226] "Starting" policy="none"
Oct 30 21:37:57.003432 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003428    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.003514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003432    4882 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Oct 30 21:37:57.003514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003454    4882 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Oct 30 21:37:57.003645 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003630    4882 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Oct 30 21:37:57.003677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003645    4882 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Oct 30 21:37:57.003677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003663    4882 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Oct 30 21:37:57.003677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003669    4882 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Oct 30 21:37:57.003677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003675    4882 policy_none.go:50] "Start"
Oct 30 21:37:57.003871 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003685    4882 memory_manager.go:187] "Starting memorymanager" policy="None"
Oct 30 21:37:57.003871 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.003697    4882 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Oct 30 21:37:57.004682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004667    4882 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Oct 30 21:37:57.004744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004684    4882 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Oct 30 21:37:57.004744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004691    4882 policy_none.go:44] "Start"
Oct 30 21:37:57.004927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004911    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.004970 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004944    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.004970 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.004959    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.009415 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.009383    4882 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Oct 30 21:37:57.012976 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.012960    4882 manager.go:341] "Starting Device Plugin manager"
Oct 30 21:37:57.013072 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.013007    4882 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Oct 30 21:37:57.013072 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013019    4882 server.go:87] "Starting device plugin registration server"
Oct 30 21:37:57.013363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013349    4882 eviction_manager.go:194] "Eviction manager: starting control loop"
Oct 30 21:37:57.013423 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013364    4882 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Oct 30 21:37:57.013538 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013524    4882 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Oct 30 21:37:57.013666 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013636    4882 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Oct 30 21:37:57.013753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013675    4882 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Oct 30 21:37:57.013753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013682    4882 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Oct 30 21:37:57.013868 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.013715    4882 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T21:37:57.013664495Z","UUID":"2fad2f21-6bdd-436c-9b39-18e6cab29f45","Handler":null,"Name":"","Endpoint":""}
Oct 30 21:37:57.018086 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.017844    4882 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Oct 30 21:37:57.018086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.017862    4882 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Oct 30 21:37:57.018086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.017891    4882 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Oct 30 21:37:57.018086 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.017908    4882 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-82.us-west-2.compute.internal\" not found"
Oct 30 21:37:57.036724 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.035940    4882 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"]
Oct 30 21:37:57.036920 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.036897    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.039435 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.039416    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.039524 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.039449    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.039524 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.039460    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.039732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.039722    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.039876 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.039859    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.040528 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040510    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.040608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040556    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.040608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040567    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.040608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040511    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.040688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040627    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.040688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040638    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.040805 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.040789    4882 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.040854 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040823    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerStarted","Data":"05da2be9e21af354c4fdeda6473b1e06353026bd4464fba5041a65e927d47acf"}
Oct 30 21:37:57.040889 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerDied","Data":"d7af3f5fc7a5c7e23cca5703413244b70b22aa32f6cdbeeaea0f9ddfafc2ac1d"}
Oct 30 21:37:57.040889 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040878    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"6b11aa0892d28c2064080199cdadd8ae","Type":"ContainerStarted","Data":"c1a469eb5c14c5486c27556b511bcc71b6df395571620c4d04aa61341265549c"}
Oct 30 21:37:57.040943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040889    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"845d50ae5087e4cd62e2c4035d1ecf3f","Type":"ContainerStarted","Data":"f2d33c453524c0fe5cb5723ca7077c32bf56ed0719c0084e45db99c8aa89f141"}
Oct 30 21:37:57.040943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040899    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" event={"ID":"845d50ae5087e4cd62e2c4035d1ecf3f","Type":"ContainerStarted","Data":"1bd8178188f7bc292755de140c33ebef95a068dfb6d2d54d2ba9145636acca1d"}
Oct 30 21:37:57.040997 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.040945    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.041691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.041673    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.041736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.041698    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.041736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.041710    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.041865 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.041853    4882 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-82.us-west-2.compute.internal\" not found" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.047183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.047166    4882 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:37:57.092774 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.092746    4882 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:37:57.114305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.114280    4882 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Oct 30 21:37:57.115299 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.115284    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientMemory"
Oct 30 21:37:57.115366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.115314    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasNoDiskPressure"
Oct 30 21:37:57.115366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.115325    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeHasSufficientPID"
Oct 30 21:37:57.115366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.115351    4882 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.138366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.138339    4882 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:37:57.140956 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.140934    4882 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Oct 30 21:37:57.185394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.185366    4882 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Oct 30 21:37:57.202822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.202797    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.235314 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.235283    4882 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.235448 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.235358    4882 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.235947 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.235930    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeNotReady"
Oct 30 21:37:57.236026 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.235963    4882 setters.go:546] "Node became not ready" node="ip-10-0-141-82.us-west-2.compute.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-10-30T21:37:57Z","lastTransitionTime":"2026-10-30T21:37:57Z","reason":"KubeletNotReady","message":"CSINode is not yet initialized"}
Oct 30 21:37:57.264688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.264664    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:37:57.264790 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.264733    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.264790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.264754    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.285808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.285769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.285931 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.285814    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.285931 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.285906    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.299171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.299113    4882 nodeinfomanager.go:471] existing CSINode "ip-10-0-141-82.us-west-2.compute.internal" is owned by different node (oldNodeID="5e9e29e8-7820-474f-b9c1-a6c488e0d42e", newNodeID=""), cleaning up...
Oct 30 21:37:57.328757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.328722    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:37:57.328890 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:57.328788    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.386840 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386795    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.386840 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386814    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.387058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386866    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.387058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386884    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.387058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386924    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/845d50ae5087e4cd62e2c4035d1ecf3f-config\") pod \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"845d50ae5087e4cd62e2c4035d1ecf3f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.387058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.386932    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6b11aa0892d28c2064080199cdadd8ae-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" (UID: \"6b11aa0892d28c2064080199cdadd8ae\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.889909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.889877    4882 apiserver.go:52] "Watching apiserver"
Oct 30 21:37:57.979799 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.979773    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.979959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.979880    4882 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:57.988395 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.988375    4882 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Oct 30 21:37:57.988842 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.988821    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal","kube-system/konnectivity-agent-sxl4l","openshift-ingress-canary/ingress-canary-6cj9n","openshift-network-diagnostics/network-check-target-fdqv5","openshift-dns/dns-default-b7bx8","openshift-insights/insights-runtime-extractor-p2prz","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b","openshift-multus/network-metrics-daemon-x465r","kube-system/global-pull-secret-syncer-6pwrc","openshift-multus/multus-additional-cni-plugins-k4kwl","openshift-multus/multus-z6crj","kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal","openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft","openshift-network-operator/iptables-alerter-pz6fg","openshift-dns/node-resolver-4cmqj","openshift-insights/periodic-gathering-7sw5w-75z9c","openshift-ovn-kubernetes/ovnkube-node-99kdf","openshift-cluster-node-tuning-operator/tuned-fgd96","openshift-service-ca/service-ca-5fffc97bdf-vrzl6","openshift-image-registry/node-ca-7rdhn"]
Oct 30 21:37:57.995510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:57.995482    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.004257 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.004232    4882 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Oct 30 21:37:58.043790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.043766    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:37:58.043899 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:58.043828    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049730    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049868    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-hcvt7\""
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049900    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p5bth\""
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049899    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zlst6\""
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049902    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Oct 30 21:37:58.049885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049951    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049965    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t84mq\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.049958    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050009    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050009    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050050    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-gfsxs\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050098    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050083    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b6pf6\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050115    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050121    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050105    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050130    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050151    4882 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050171    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-hkmjr\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050177    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050152    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-j5bph\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050176    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050225    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050285    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050316    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050361    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ktl9g\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050433    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050454    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-lt5wb\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050499    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Oct 30 21:37:58.050544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050555    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-v9xch\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050615    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050695    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-x7gkt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050777    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050791    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050874    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050881    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050935    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.050952    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:37:58.050977    4882 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-82.us-west-2.compute.internal"
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051075    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051084    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051088    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051084    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051109    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051210    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-cb4gr\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051314    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Oct 30 21:37:58.051565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051490    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.052028 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.051797    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.078640 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078614    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.078836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078661    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.078836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078679    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Oct 30 21:37:58.078836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078697    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Oct 30 21:37:58.078836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078762    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.078836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078768    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Oct 30 21:37:58.079061 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.078948    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.079061 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.079013    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.079154 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.079104    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.079154 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.079112    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Oct 30 21:37:58.090431 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.090498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090438    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.090498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090471    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65tq4\" (UniqueName: \"kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.090579 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090496    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.090579 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090540    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5jns\" (UniqueName: \"kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.090579 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.090720 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090701    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.090786 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090740    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67xtz\" (UniqueName: \"kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.090786 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.090891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090786    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-script-lib\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.090891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090796    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.090891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090828    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090922    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090926    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-cabundle\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090963    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.090995    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091019    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.091060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091043    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091061    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091095    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091122    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091142    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djmm6\" (UniqueName: \"kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091159    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091182    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091207    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091230    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091297    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091338    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw5h\" (UniqueName: \"kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091367    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091378    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cni-binary-copy\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091428    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/2bf2774c-7d35-4bff-98ae-489cc598b066-agent-certs\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091438    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqjqw\" (UniqueName: \"kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091464    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091509    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/07f98edf-bb0a-422e-ab8f-c1f935774f2a-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091538    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091562    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091596    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091654    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091653    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091688    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091718    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9k26r\" (UniqueName: \"kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.091827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091733    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091844    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091871    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091894    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091923    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h7pqv\" (UniqueName: \"kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.091959    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092140    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092192    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-tmp-dir\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092253    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092273    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092287    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092303    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092318    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092363    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.092399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092432    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tq8jv\" (UniqueName: \"kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092435    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ef440513-6ad3-4663-92fb-1b69c378c66f-tmp-dir\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092448    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092481    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092601    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-daemon-config\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092622    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ggqtk\" (UniqueName: \"kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092665    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092680    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092696    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cni-binary-copy\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092707    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092771    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/abb604c9-145e-4bdb-8679-a5a0c25b1441-data-volume\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092774    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btqkw\" (UniqueName: \"kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092811    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092926    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092957    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092975    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovn-node-metrics-cert\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.092988    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/2bf2774c-7d35-4bff-98ae-489cc598b066-konnectivity-ca\") pod \"konnectivity-agent-sxl4l\" (UID: \"2bf2774c-7d35-4bff-98ae-489cc598b066\") " pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvgsw\" (UniqueName: \"kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093082    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093157    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-tmp\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093160    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.093241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093238    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093266    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093277    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/f0d8c917-e87e-4304-bec9-66f213536b88-serviceca\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093292    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093318    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093343    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/8ee90e5a-7c9f-4b24-841c-86b50175efd6-iptables-alerter-script\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093349    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jw5z\" (UniqueName: \"kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093387    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093428    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093445    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093481    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093509    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/ef440513-6ad3-4663-92fb-1b69c378c66f-metrics-tls\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093520    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093548    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f1d1b936-7975-413e-8b25-ace2bd71cdd3-metrics-certs\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093605    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfsf7\" (UniqueName: \"kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7\") pod \"migrator-66df5d496b-6n6ft\" (UID: \"2328abe6-8360-4e86-9a59-4e308704aac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093624    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-env-overrides\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093635    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093657    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093675    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093690    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093708    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d97ac33e-24e8-4256-a3cd-154231e10553-original-pull-secret\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093828    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093873    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093888    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-ovnkube-config\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093922    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-tuned\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093959    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.093989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094010    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094087    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094121    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094150    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094193    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094230    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.094379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094252    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.095523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.094433    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/ef440513-6ad3-4663-92fb-1b69c378c66f-config-volume\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.096777 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096728    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-4rnkh\""
Oct 30 21:37:58.096777 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096746    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-phtlh\""
Oct 30 21:37:58.096777 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096758    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Oct 30 21:37:58.096777 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096758    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Oct 30 21:37:58.097001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096728    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Oct 30 21:37:58.097001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096856    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-nc2jc\""
Oct 30 21:37:58.097001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.096909    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-2fnrw\""
Oct 30 21:37:58.097136 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.097004    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-xlcqp\""
Oct 30 21:37:58.097136 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.097089    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Oct 30 21:37:58.097454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.097438    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Oct 30 21:37:58.101710 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.101694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5da833b-0939-4f20-a2b3-18fe731a9360-signing-key\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.103170 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.103150    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.103984 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.103967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-cert\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:58.157090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.157010    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5jns\" (UniqueName: \"kubernetes.io/projected/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-kube-api-access-h5jns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.159597 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.159570    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65tq4\" (UniqueName: \"kubernetes.io/projected/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-kube-api-access-65tq4\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.160631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.160615    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h7pqv\" (UniqueName: \"kubernetes.io/projected/a1a38f40-6f73-4d05-9ca4-d82518e4bbe7-kube-api-access-h7pqv\") pod \"ingress-canary-6cj9n\" (UID: \"a1a38f40-6f73-4d05-9ca4-d82518e4bbe7\") " pod="openshift-ingress-canary/ingress-canary-6cj9n"
Oct 30 21:37:58.162661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.162638    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw5h\" (UniqueName: \"kubernetes.io/projected/07f98edf-bb0a-422e-ab8f-c1f935774f2a-kube-api-access-fvw5h\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.162789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.162764    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djmm6\" (UniqueName: \"kubernetes.io/projected/f1d1b936-7975-413e-8b25-ace2bd71cdd3-kube-api-access-djmm6\") pod \"network-metrics-daemon-x465r\" (UID: \"f1d1b936-7975-413e-8b25-ace2bd71cdd3\") " pod="openshift-multus/network-metrics-daemon-x465r"
Oct 30 21:37:58.163488 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.163467    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tq8jv\" (UniqueName: \"kubernetes.io/projected/8ee90e5a-7c9f-4b24-841c-86b50175efd6-kube-api-access-tq8jv\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.163760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.163744    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jw5z\" (UniqueName: \"kubernetes.io/projected/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-kube-api-access-6jw5z\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.164601 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.164569    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfsf7\" (UniqueName: \"kubernetes.io/projected/2328abe6-8360-4e86-9a59-4e308704aac1-kube-api-access-mfsf7\") pod \"migrator-66df5d496b-6n6ft\" (UID: \"2328abe6-8360-4e86-9a59-4e308704aac1\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-6n6ft"
Oct 30 21:37:58.164681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.164661    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vgn88\" (UniqueName: \"kubernetes.io/projected/3b4f2e52-d72d-4962-9627-e1817b1774a9-kube-api-access-vgn88\") pod \"network-check-target-fdqv5\" (UID: \"3b4f2e52-d72d-4962-9627-e1817b1774a9\") " pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:37:58.164873 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.164854    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvgsw\" (UniqueName: \"kubernetes.io/projected/abb604c9-145e-4bdb-8679-a5a0c25b1441-kube-api-access-vvgsw\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.165232 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.165201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btqkw\" (UniqueName: \"kubernetes.io/projected/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-kube-api-access-btqkw\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.165466 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.165452    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ggqtk\" (UniqueName: \"kubernetes.io/projected/ef440513-6ad3-4663-92fb-1b69c378c66f-kube-api-access-ggqtk\") pod \"dns-default-b7bx8\" (UID: \"ef440513-6ad3-4663-92fb-1b69c378c66f\") " pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.165540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.165524    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67xtz\" (UniqueName: \"kubernetes.io/projected/79565b90-0624-4317-983b-6f44b73c6f78-kube-api-access-67xtz\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.168262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.168244    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"periodic-gathering-7sw5w-75z9c\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") " pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:37:58.168356 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.168341    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqjqw\" (UniqueName: \"kubernetes.io/projected/b5da833b-0939-4f20-a2b3-18fe731a9360-kube-api-access-fqjqw\") pod \"service-ca-5fffc97bdf-vrzl6\" (UID: \"b5da833b-0939-4f20-a2b3-18fe731a9360\") " pod="openshift-service-ca/service-ca-5fffc97bdf-vrzl6"
Oct 30 21:37:58.170527 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.170508    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9k26r\" (UniqueName: \"kubernetes.io/projected/f0d8c917-e87e-4304-bec9-66f213536b88-kube-api-access-9k26r\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.195499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195473    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.195558 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195502    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.195558 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195517    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.195558 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195532    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195603    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-var-lib-kubelet\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195615    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-systemd\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195617    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-etc-selinux\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195617    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195647    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-kubelet-config\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195662    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-node-log\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195668    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195701    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-system-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.195728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195712    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195737    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195761    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195773    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-system-cni-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195787    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195795    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-etc-kubernetes\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195811    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-cnibin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195810    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195843    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-conf-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195820    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-k8s-cni-cncf-io\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195855    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195905    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195907    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-host\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195930    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195962    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195966    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.195978    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196009    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196034    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-run-netns\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196037    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196035    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-socket-dir-parent\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196047    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-multus-cni-dir\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196066    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-kubelet\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196066    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196101    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196071    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/abb604c9-145e-4bdb-8679-a5a0c25b1441-crio-socket\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196108    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/4d6a00c1-5c3d-48e9-974d-9ac8477c2f79-hosts-file\") pod \"node-resolver-4cmqj\" (UID: \"4d6a00c1-5c3d-48e9-974d-9ac8477c2f79\") " pod="openshift-dns/node-resolver-4cmqj"
Oct 30 21:37:58.196098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196118    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196038    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d97ac33e-24e8-4256-a3cd-154231e10553-dbus\") pod \"global-pull-secret-syncer-6pwrc\" (UID: \"d97ac33e-24e8-4256-a3cd-154231e10553\") " pod="kube-system/global-pull-secret-syncer-6pwrc"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196138    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-bin\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196160    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-run\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196168    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196191    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196220    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-etc-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196233    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196248    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196268    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196293    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196306    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-device-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196307    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-kubernetes\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196338    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-socket-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196356    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-registration-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196389    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196412    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196411    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196448    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-var-lib-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196455    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysconfig\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196484    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196512    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196534    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-openvswitch\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196536    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196563    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-multus\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196557    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196607    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196606    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196621    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-systemd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196637    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196647    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-kubelet-dir\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196662    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196672    4882 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196702    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f0d8c917-e87e-4304-bec9-66f213536b88-host\") pod \"node-ca-7rdhn\" (UID: \"f0d8c917-e87e-4304-bec9-66f213536b88\") " pod="openshift-image-registry/node-ca-7rdhn"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196708    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/8ee90e5a-7c9f-4b24-841c-86b50175efd6-host-slash\") pod \"iptables-alerter-pz6fg\" (UID: \"8ee90e5a-7c9f-4b24-841c-86b50175efd6\") " pod="openshift-network-operator/iptables-alerter-pz6fg"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196726    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196696    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-systemd-units\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196766    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196795    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196814    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196827    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-log-socket\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196835    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-netns\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196831    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196855    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-sys\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-run-multus-certs\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196868    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196936    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196950    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196963    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196963    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-cni-netd\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.196956    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-tuning-conf-dir\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.197002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197029    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/79565b90-0624-4317-983b-6f44b73c6f78-sys-fs\") pod \"aws-ebs-csi-driver-node-b8h4b\" (UID: \"79565b90-0624-4317-983b-6f44b73c6f78\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-b8h4b"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197053    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-host-var-lib-cni-bin\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197094    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197128    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-modprobe-d\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197152    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197165    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197178    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197192    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-etc-sysctl-conf\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197204    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197250    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-kubelet\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197257    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/1e474a6c-d260-4e3e-bdb9-d3a00d5e3749-hostroot\") pod \"multus-z6crj\" (UID: \"1e474a6c-d260-4e3e-bdb9-d3a00d5e3749\") " pod="openshift-multus/multus-z6crj"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197286    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197303    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/07f98edf-bb0a-422e-ab8f-c1f935774f2a-cnibin\") pod \"multus-additional-cni-plugins-k4kwl\" (UID: \"07f98edf-bb0a-422e-ab8f-c1f935774f2a\") " pod="openshift-multus/multus-additional-cni-plugins-k4kwl"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197313    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e021af5f-e7c2-474e-aff2-7cc532fbcaf0-lib-modules\") pod \"tuned-fgd96\" (UID: \"e021af5f-e7c2-474e-aff2-7cc532fbcaf0\") " pod="openshift-cluster-node-tuning-operator/tuned-fgd96"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197349    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197368    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-run-ovn\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.198478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.197391    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa-host-slash\") pod \"ovnkube-node-99kdf\" (UID: \"71ef1ace-7a0c-4e16-9b99-2cc43e7ebaaa\") " pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:37:58.199199 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.199184    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/abb604c9-145e-4bdb-8679-a5a0c25b1441-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-p2prz\" (UID: \"abb604c9-145e-4bdb-8679-a5a0c25b1441\") " pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.296354 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.296323    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-p2prz"
Oct 30 21:37:58.659883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.659752    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-p2prz"]
Oct 30 21:37:58.691766 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:37:58.691738    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabb604c9_145e_4bdb_8679_a5a0c25b1441.slice/crio-f031ddccb1e6b943af12115dbe97752eb032061e9e842de37fac3cb0261a6c19 WatchSource:0}: Error finding container f031ddccb1e6b943af12115dbe97752eb032061e9e842de37fac3cb0261a6c19: Status 404 returned error can't find the container with id f031ddccb1e6b943af12115dbe97752eb032061e9e842de37fac3cb0261a6c19
Oct 30 21:37:58.744903 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.744882    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:37:58.768772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.768743    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-b7bx8"
Oct 30 21:37:58.984708 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.984612    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2prz" event={"ID":"abb604c9-145e-4bdb-8679-a5a0c25b1441","Type":"ContainerStarted","Data":"8f549e7566b5915350974e0bc505ca06dce1b52c0134905f1ce2b712b62a1a04"}
Oct 30 21:37:58.984708 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:58.984672    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2prz" event={"ID":"abb604c9-145e-4bdb-8679-a5a0c25b1441","Type":"ContainerStarted","Data":"f031ddccb1e6b943af12115dbe97752eb032061e9e842de37fac3cb0261a6c19"}
Oct 30 21:37:59.227109 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:37:59.227082    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:38:00.997095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:00.997059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2prz" event={"ID":"abb604c9-145e-4bdb-8679-a5a0c25b1441","Type":"ContainerStarted","Data":"4804199b54ec8345e7cdb029f2c45ccb093018eb26dcc953ae11b08b1959b905"}
Oct 30 21:38:03.005580 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:03.005544    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-p2prz" event={"ID":"abb604c9-145e-4bdb-8679-a5a0c25b1441","Type":"ContainerStarted","Data":"36fffb2fa8f31b133a04e3b4bb10b84910cb3c20098415ec2b79932460a8b96d"}
Oct 30 21:38:03.302533 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:03.302448    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:38:03.802539 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:03.802506    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:38:03.802764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:03.802750    4882 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Oct 30 21:38:07.514564 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:07.514478    4882 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-82.us-west-2.compute.internal" event="NodeReady"
Oct 30 21:38:11.075240 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.075202    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"]
Oct 30 21:38:11.099845 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.099815    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.134866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.134833    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"]
Oct 30 21:38:11.175903 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.175867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.176077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.175913    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.176077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.175984    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361370c9-679d-4243-9450-7a315ec4f459-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.176145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.176095    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9dv2\" (UniqueName: \"kubernetes.io/projected/361370c9-679d-4243-9450-7a315ec4f459-kube-api-access-v9dv2\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.180407 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.180387    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Oct 30 21:38:11.199233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.199206    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Oct 30 21:38:11.199383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.199241    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:38:11.199383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.199255    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Oct 30 21:38:11.199383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.199241    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-d9jhk\""
Oct 30 21:38:11.199383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.199302    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Oct 30 21:38:11.228220 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.228187    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"]
Oct 30 21:38:11.230404 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.230380    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-6b7ms"]
Oct 30 21:38:11.230531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.230498    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.232512 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.232493    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.266563 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.266527    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"]
Oct 30 21:38:11.276692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276664    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-textfile\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.276836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276699    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-root\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.276836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276726    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-tls\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.276836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276814    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.276971 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276881    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.276971 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276916    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.276971 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276940    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.277113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.276994    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.277113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277067    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-metrics-client-ca\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277102    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9dv2\" (UniqueName: \"kubernetes.io/projected/361370c9-679d-4243-9450-7a315ec4f459-kube-api-access-v9dv2\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.277247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277146    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4sr6\" (UniqueName: \"kubernetes.io/projected/773ef005-be9c-4f87-b282-c5fad993d143-kube-api-access-x4sr6\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.277247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277172    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-wtmp\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277192    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.277247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277211    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277239    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/773ef005-be9c-4f87-b282-c5fad993d143-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.277438 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277287    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-sys\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277438 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277321    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361370c9-679d-4243-9450-7a315ec4f459-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.277438 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277417    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kbf\" (UniqueName: \"kubernetes.io/projected/0ab31f1a-cea7-4422-83e6-d146cb9f920a-kube-api-access-v2kbf\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277557 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277458    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.277975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.277947    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/361370c9-679d-4243-9450-7a315ec4f459-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.279302 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.279281    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.279509 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.279490    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/361370c9-679d-4243-9450-7a315ec4f459-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.281636 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.281618    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Oct 30 21:38:11.281636 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.281644    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-r56fv\""
Oct 30 21:38:11.281937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.281921    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-zf5wc\""
Oct 30 21:38:11.282004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.281990    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Oct 30 21:38:11.282086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.282069    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Oct 30 21:38:11.282577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.282563    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Oct 30 21:38:11.305470 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.305445    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Oct 30 21:38:11.305598 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.305508    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Oct 30 21:38:11.335568 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.335539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9dv2\" (UniqueName: \"kubernetes.io/projected/361370c9-679d-4243-9450-7a315ec4f459-kube-api-access-v9dv2\") pod \"openshift-state-metrics-7bd7489957-q4h9q\" (UID: \"361370c9-679d-4243-9450-7a315ec4f459\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.377997 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.377965    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kbf\" (UniqueName: \"kubernetes.io/projected/0ab31f1a-cea7-4422-83e6-d146cb9f920a-kube-api-access-v2kbf\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378006    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378038    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-textfile\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378063    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-root\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378088    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-tls\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378114    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378145    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378181 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378180    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-root\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378200    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378244    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-metrics-client-ca\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378294    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4sr6\" (UniqueName: \"kubernetes.io/projected/773ef005-be9c-4f87-b282-c5fad993d143-kube-api-access-x4sr6\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378322    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-wtmp\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378351    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378373    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/773ef005-be9c-4f87-b282-c5fad993d143-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378397    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-sys\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378436    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-textfile\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.378516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378490    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-sys\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.379041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378626    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-wtmp\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.379041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378819    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-accelerators-collector-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.379041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378971    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.379041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.378971    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/773ef005-be9c-4f87-b282-c5fad993d143-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.379260 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.379158    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0ab31f1a-cea7-4422-83e6-d146cb9f920a-metrics-client-ca\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.379315 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.379298    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/773ef005-be9c-4f87-b282-c5fad993d143-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.384498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.381000    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.384498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.381064    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/0ab31f1a-cea7-4422-83e6-d146cb9f920a-node-exporter-tls\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.384498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.381186    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.384498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.381375    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/773ef005-be9c-4f87-b282-c5fad993d143-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.412850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.412815    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"
Oct 30 21:38:11.459772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.459742    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kbf\" (UniqueName: \"kubernetes.io/projected/0ab31f1a-cea7-4422-83e6-d146cb9f920a-kube-api-access-v2kbf\") pod \"node-exporter-6b7ms\" (UID: \"0ab31f1a-cea7-4422-83e6-d146cb9f920a\") " pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.460040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.460019    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4sr6\" (UniqueName: \"kubernetes.io/projected/773ef005-be9c-4f87-b282-c5fad993d143-kube-api-access-x4sr6\") pod \"kube-state-metrics-5dc8d7d89b-7pfdn\" (UID: \"773ef005-be9c-4f87-b282-c5fad993d143\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.544312 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.544272    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"
Oct 30 21:38:11.548855 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.548831    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-6b7ms"
Oct 30 21:38:11.561265 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:11.561232    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ab31f1a_cea7_4422_83e6_d146cb9f920a.slice/crio-89881f637f8ae4bdc5174d94351c4893f035c0002cae3aac3936f38d96a09077 WatchSource:0}: Error finding container 89881f637f8ae4bdc5174d94351c4893f035c0002cae3aac3936f38d96a09077: Status 404 returned error can't find the container with id 89881f637f8ae4bdc5174d94351c4893f035c0002cae3aac3936f38d96a09077
Oct 30 21:38:11.829250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.829219    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q"]
Oct 30 21:38:11.831643 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:11.831615    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod361370c9_679d_4243_9450_7a315ec4f459.slice/crio-1080c3c101b7017621bd2289a3f8b4b39c47251f1c03bdfcaadf89b2923f5063 WatchSource:0}: Error finding container 1080c3c101b7017621bd2289a3f8b4b39c47251f1c03bdfcaadf89b2923f5063: Status 404 returned error can't find the container with id 1080c3c101b7017621bd2289a3f8b4b39c47251f1c03bdfcaadf89b2923f5063
Oct 30 21:38:11.922332 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:11.922187    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn"]
Oct 30 21:38:11.924905 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:11.924880    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod773ef005_be9c_4f87_b282_c5fad993d143.slice/crio-5c50a2056d59c50a76edbec2b21b2013ac81ab1d0462d4fcd9abc72ec542cc6b WatchSource:0}: Error finding container 5c50a2056d59c50a76edbec2b21b2013ac81ab1d0462d4fcd9abc72ec542cc6b: Status 404 returned error can't find the container with id 5c50a2056d59c50a76edbec2b21b2013ac81ab1d0462d4fcd9abc72ec542cc6b
Oct 30 21:38:12.041114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:12.041080    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerStarted","Data":"5c50a2056d59c50a76edbec2b21b2013ac81ab1d0462d4fcd9abc72ec542cc6b"}
Oct 30 21:38:12.042906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:12.042875    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q" event={"ID":"361370c9-679d-4243-9450-7a315ec4f459","Type":"ContainerStarted","Data":"250b24157d8eee8e2619cdcd5263c577beebefe617833b7b51a371a0190ca7d3"}
Oct 30 21:38:12.043050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:12.042928    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q" event={"ID":"361370c9-679d-4243-9450-7a315ec4f459","Type":"ContainerStarted","Data":"acfe16845ac38e9c3b176b5637bd2e9d3c23b76d5660da8053b5ffeee66b22ae"}
Oct 30 21:38:12.043050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:12.042953    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q" event={"ID":"361370c9-679d-4243-9450-7a315ec4f459","Type":"ContainerStarted","Data":"1080c3c101b7017621bd2289a3f8b4b39c47251f1c03bdfcaadf89b2923f5063"}
Oct 30 21:38:12.044019 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:12.043997    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6b7ms" event={"ID":"0ab31f1a-cea7-4422-83e6-d146cb9f920a","Type":"ContainerStarted","Data":"89881f637f8ae4bdc5174d94351c4893f035c0002cae3aac3936f38d96a09077"}
Oct 30 21:38:13.183821 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.183789    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:38:13.186390 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.186366    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.190731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190704    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.190888 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190750    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.190888 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190859    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.191014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190930    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsvzb\" (UniqueName: \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.191014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190962    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.191014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190983    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.191014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.190998    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.191207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.191020    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.238607 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.238562    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-czbd6\""
Oct 30 21:38:13.243955 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.243930    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Oct 30 21:38:13.244133 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.244063    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Oct 30 21:38:13.244210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.244137    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Oct 30 21:38:13.244210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.244194    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Oct 30 21:38:13.244435 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.244417    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Oct 30 21:38:13.246217 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.246191    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:38:13.246337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.246324    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Oct 30 21:38:13.246399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.246372    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298530    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298635    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsvzb\" (UniqueName: \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298720    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298751    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298778    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298820    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.298866    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.299974    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.300659    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.300821 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.300775    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.301498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.301450    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.302869 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.302838    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.304514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.304490    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.345570 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.345522    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Oct 30 21:38:13.352405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.352372    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.360645 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.360615    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsvzb\" (UniqueName: \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\") pod \"console-5c5dd65c67-2pfcs\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") " pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.499988 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.499958    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:13.500184 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.500094    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:13.504760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.504734    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.562840    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.562917    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.562848    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.562929    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.563129    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.563195    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:38:13.563303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.563270    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:38:13.563680 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.563664    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fj7sr\""
Oct 30 21:38:13.563772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.563759    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:38:13.564144 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.564127    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:38:13.569111 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.569022    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:13.600697 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600656    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.600904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600713    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.600904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600744    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d8sbs\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.600904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600866    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601052 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600902    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601052 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.600941    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601052 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601019    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601069    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601104    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601135    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601154    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601186    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.601250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.601215    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.699921 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.699889    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-99kdf"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702005    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702055    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702236    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d8sbs\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702289    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702318    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702356    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702407    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702437    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702471    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702518    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702543    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702572    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702620    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.703327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.702925    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.704160 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.703744    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.704160 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.703794    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.706964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.706934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.706964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.706963    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.707400 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.707377    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.707535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.707395    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.707820 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.707797    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.707898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.707864    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.708487 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.708431    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.708581 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.708543    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.709740 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.709689    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.760747 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.760715    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d8sbs\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\") pod \"alertmanager-main-0\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.841728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.841689    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:13.864540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.864510    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"]
Oct 30 21:38:13.869596 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.869558    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.879918 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.879888    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904090    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904148    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-grpc-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904181    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-metrics-client-ca\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904241    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fsp64\" (UniqueName: \"kubernetes.io/projected/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-kube-api-access-fsp64\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904282    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904315    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904353    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.905231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.904447    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-tmp\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:13.922190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922116    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Oct 30 21:38:13.922190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922136    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Oct 30 21:38:13.922190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922131    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Oct 30 21:38:13.922445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922285    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Oct 30 21:38:13.922445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922362    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Oct 30 21:38:13.922445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.922377    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-7kfbm\""
Oct 30 21:38:13.923105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.923088    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-7nh62kpibo0o4\""
Oct 30 21:38:13.931218 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:13.931186    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"]
Oct 30 21:38:14.005803 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005776    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.005934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.005934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005898    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-tmp\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.005934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005931    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.006105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005959    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-grpc-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.006105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.005979    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-metrics-client-ca\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.006105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.006004    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fsp64\" (UniqueName: \"kubernetes.io/projected/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-kube-api-access-fsp64\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.006105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.006026    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.006105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.006042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.007295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.007264    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-tmp\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.007436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.007367    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-metrics-client-ca\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.008703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.008675    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.008855 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.008829    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.009016 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.008992    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-grpc-tls\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.009926 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.009900    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.010081 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.010059    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.010148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.010101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.055448 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.055404    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerStarted","Data":"8db23c878948cca4fe36a8f0f57ef69a25bf02282875c75091b7bcd846174232"}
Oct 30 21:38:14.057665 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.057628    4882 generic.go:356] "Generic (PLEG): container finished" podID="0ab31f1a-cea7-4422-83e6-d146cb9f920a" containerID="6617d96ac2fe55327cfa5d292e6dadaab9714ed1de8f0886a99f48f8d0a92e74" exitCode=0
Oct 30 21:38:14.057803 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.057712    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6b7ms" event={"ID":"0ab31f1a-cea7-4422-83e6-d146cb9f920a","Type":"ContainerDied","Data":"6617d96ac2fe55327cfa5d292e6dadaab9714ed1de8f0886a99f48f8d0a92e74"}
Oct 30 21:38:14.059658 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.059624    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5dd65c67-2pfcs" event={"ID":"d5c91726-0d2c-40f0-a0a3-ecfe3830def9","Type":"ContainerStarted","Data":"41b919e1de8511ba47c4104bedf2aa1c350915f477f5c87409ab86798bed5012"}
Oct 30 21:38:14.065610 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.065486    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fsp64\" (UniqueName: \"kubernetes.io/projected/2024b1bc-1853-477d-8ae2-8329bd3dbb8f-kube-api-access-fsp64\") pod \"thanos-querier-fc58c9df8-bjk8x\" (UID: \"2024b1bc-1853-477d-8ae2-8329bd3dbb8f\") " pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.200126 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.200085    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:14.280111 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.279725    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:14.282011 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:14.281958    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7241e2f5_2bda_4563_b53f_2fb03e9fb4f9.slice/crio-77db7289b6003f430a95936b70983e11bbcd28f6564ea2ac0733719d2a33b6d2 WatchSource:0}: Error finding container 77db7289b6003f430a95936b70983e11bbcd28f6564ea2ac0733719d2a33b6d2: Status 404 returned error can't find the container with id 77db7289b6003f430a95936b70983e11bbcd28f6564ea2ac0733719d2a33b6d2
Oct 30 21:38:14.574634 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:14.574478    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"]
Oct 30 21:38:14.577294 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:14.577265    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2024b1bc_1853_477d_8ae2_8329bd3dbb8f.slice/crio-09a9b2866d4fa8e2872edc457853778e258aa530677efc3e29f68ada5853aa11 WatchSource:0}: Error finding container 09a9b2866d4fa8e2872edc457853778e258aa530677efc3e29f68ada5853aa11: Status 404 returned error can't find the container with id 09a9b2866d4fa8e2872edc457853778e258aa530677efc3e29f68ada5853aa11
Oct 30 21:38:15.065700 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.065649    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerStarted","Data":"878705150b46d8cb6662734aae9368813ab91b1d463f130dfc04e74ca0080d38"}
Oct 30 21:38:15.065902 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.065737    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerStarted","Data":"9db5ac30edfef67306b80b0808d9ef3c9c1e62113200fb213d9190edd930b328"}
Oct 30 21:38:15.068219 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.068191    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q" event={"ID":"361370c9-679d-4243-9450-7a315ec4f459","Type":"ContainerStarted","Data":"4ab9709477a459b0d827c8643a3930f60611a177862a6855cfad2166d3e0c7f9"}
Oct 30 21:38:15.071066 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.071041    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6b7ms" event={"ID":"0ab31f1a-cea7-4422-83e6-d146cb9f920a","Type":"ContainerStarted","Data":"29375206f48ec4b7c97340f6cfa6f92c4e2c205782c309879f77ae24116f55a8"}
Oct 30 21:38:15.071196 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.071089    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-6b7ms" event={"ID":"0ab31f1a-cea7-4422-83e6-d146cb9f920a","Type":"ContainerStarted","Data":"ba2deb1084b16852f56a9acd46e4d82328899a7418fb603dd05586c4e9526132"}
Oct 30 21:38:15.072388 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.072358    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"09a9b2866d4fa8e2872edc457853778e258aa530677efc3e29f68ada5853aa11"}
Oct 30 21:38:15.073719 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.073693    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"77db7289b6003f430a95936b70983e11bbcd28f6564ea2ac0733719d2a33b6d2"}
Oct 30 21:38:15.180602 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.180505    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" podStartSLOduration=2.167559592 podStartE2EDuration="4.180482218s" totalImagesPullingTime="2.012922626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.17890036 +0000 UTC m=+18.352224788" watchObservedRunningTime="2026-10-30 21:38:15.180482218 +0000 UTC m=+18.353806644"
Oct 30 21:38:15.292978 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.292919    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-q4h9q" podStartSLOduration=1.79524847 podStartE2EDuration="4.292901433s" totalImagesPullingTime="2.497652963s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.291571424 +0000 UTC m=+18.464895843" watchObservedRunningTime="2026-10-30 21:38:15.292901433 +0000 UTC m=+18.466225858"
Oct 30 21:38:15.408227 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.408170    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-6b7ms" podStartSLOduration=2.428232725 podStartE2EDuration="4.408151156s" totalImagesPullingTime="1.889185814s" totalInitContainerRuntime="90.732617ms" isStatefulPod=true podCreationTimestamp="2026-10-30 21:38:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:15.40737111 +0000 UTC m=+18.580695536" watchObservedRunningTime="2026-10-30 21:38:15.408151156 +0000 UTC m=+18.581475582"
Oct 30 21:38:15.911180 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.911141    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"]
Oct 30 21:38:15.914540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.914494    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:15.967764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.967730    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Oct 30 21:38:15.967959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.967839    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-lww2l\""
Oct 30 21:38:15.980084 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:15.980052    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"]
Oct 30 21:38:16.022833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.022806    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9z48z\" (UniqueName: \"kubernetes.io/projected/7bb75240-abd5-43e1-aa11-7604abed835d-kube-api-access-9z48z\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.022981 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.022882    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bb75240-abd5-43e1-aa11-7604abed835d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.085113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.085083    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"}
Oct 30 21:38:16.100012 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.099982    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-767dd57b75-mqc9s"]
Oct 30 21:38:16.104046 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.103668    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.123398 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.123364    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5krd\" (UniqueName: \"kubernetes.io/projected/33313b40-bb78-46ad-8229-08f46def4870-kube-api-access-w5krd\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.123572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.123456    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-metrics-server-audit-profiles\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.123572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.123481    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-client-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.123572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.123536    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.124813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.124142    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9z48z\" (UniqueName: \"kubernetes.io/projected/7bb75240-abd5-43e1-aa11-7604abed835d-kube-api-access-9z48z\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.124952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.124917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33313b40-bb78-46ad-8229-08f46def4870-audit-log\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.125924 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.125882    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-client-certs\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.126188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.126165    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-tls\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.126268 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.126245    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bb75240-abd5-43e1-aa11-7604abed835d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.129935 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.129911    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7bb75240-abd5-43e1-aa11-7604abed835d-monitoring-plugin-cert\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.156897 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.156864    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Oct 30 21:38:16.157048 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.156965    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-86nh1vsdmgb09\""
Oct 30 21:38:16.161409 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.161333    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Oct 30 21:38:16.166819 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.166786    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Oct 30 21:38:16.169195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.169133    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767dd57b75-mqc9s"]
Oct 30 21:38:16.191382 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.191350    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9z48z\" (UniqueName: \"kubernetes.io/projected/7bb75240-abd5-43e1-aa11-7604abed835d-kube-api-access-9z48z\") pod \"monitoring-plugin-77f4f76594-fnwfz\" (UID: \"7bb75240-abd5-43e1-aa11-7604abed835d\") " pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.227445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227408    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5krd\" (UniqueName: \"kubernetes.io/projected/33313b40-bb78-46ad-8229-08f46def4870-kube-api-access-w5krd\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227466    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-metrics-server-audit-profiles\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227494    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-client-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227525    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227566    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33313b40-bb78-46ad-8229-08f46def4870-audit-log\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227857 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227629    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-client-certs\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.227857 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.227660    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-tls\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.229242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.228364    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/33313b40-bb78-46ad-8229-08f46def4870-audit-log\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.229242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.228383    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.229242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.229201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/33313b40-bb78-46ad-8229-08f46def4870-metrics-server-audit-profiles\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.231287 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.230641    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:16.231287 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.231244    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-tls\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.231957 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.231939    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-client-ca-bundle\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.259512 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.259461    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Oct 30 21:38:16.259681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.259626    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-8m459\""
Oct 30 21:38:16.272618 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.272530    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/33313b40-bb78-46ad-8229-08f46def4870-secret-metrics-server-client-certs\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.287270 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.287217    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5krd\" (UniqueName: \"kubernetes.io/projected/33313b40-bb78-46ad-8229-08f46def4870-kube-api-access-w5krd\") pod \"metrics-server-767dd57b75-mqc9s\" (UID: \"33313b40-bb78-46ad-8229-08f46def4870\") " pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.448972 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.448892    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:16.614900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.614861    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"]
Oct 30 21:38:16.618865 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:16.618837    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bb75240_abd5_43e1_aa11_7604abed835d.slice/crio-8a895f806657e15b4fff0e1c9f81b06527aae346255ad8228af0cf417e955a27 WatchSource:0}: Error finding container 8a895f806657e15b4fff0e1c9f81b06527aae346255ad8228af0cf417e955a27: Status 404 returned error can't find the container with id 8a895f806657e15b4fff0e1c9f81b06527aae346255ad8228af0cf417e955a27
Oct 30 21:38:16.832110 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.832081    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-767dd57b75-mqc9s"]
Oct 30 21:38:16.836648 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:16.836618    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33313b40_bb78_46ad_8229_08f46def4870.slice/crio-8c9541294502e5acb47e301416bb3a3fda28478a8a14ac0706088b049ae0e888 WatchSource:0}: Error finding container 8c9541294502e5acb47e301416bb3a3fda28478a8a14ac0706088b049ae0e888: Status 404 returned error can't find the container with id 8c9541294502e5acb47e301416bb3a3fda28478a8a14ac0706088b049ae0e888
Oct 30 21:38:16.984859 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.984823    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"]
Oct 30 21:38:16.991521 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:16.991490    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035487 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035455    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035500    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qztb\" (UniqueName: \"kubernetes.io/projected/479b5a03-a117-4edf-8da2-92f43a614802-kube-api-access-6qztb\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035543    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035713    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-metrics-client-ca\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-federate-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035843    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.035909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.035898    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-serving-certs-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.041822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.041791    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Oct 30 21:38:17.042129 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.042094    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Oct 30 21:38:17.042228 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.042143    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Oct 30 21:38:17.042305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.042283    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-wqxc2\""
Oct 30 21:38:17.042371 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.042325    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Oct 30 21:38:17.043490 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.043465    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Oct 30 21:38:17.043620 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.043566    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Oct 30 21:38:17.055046 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.055012    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"]
Oct 30 21:38:17.090514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.090423    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz" event={"ID":"7bb75240-abd5-43e1-aa11-7604abed835d","Type":"ContainerStarted","Data":"8a895f806657e15b4fff0e1c9f81b06527aae346255ad8228af0cf417e955a27"}
Oct 30 21:38:17.094128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.094096    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7" exitCode=0
Oct 30 21:38:17.094295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.094216    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"}
Oct 30 21:38:17.096150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.096117    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s" event={"ID":"33313b40-bb78-46ad-8229-08f46def4870","Type":"ContainerStarted","Data":"8c9541294502e5acb47e301416bb3a3fda28478a8a14ac0706088b049ae0e888"}
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136595    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136640    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-serving-certs-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136671    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136702    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qztb\" (UniqueName: \"kubernetes.io/projected/479b5a03-a117-4edf-8da2-92f43a614802-kube-api-access-6qztb\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136735    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136777    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136869    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-metrics-client-ca\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.137278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.136886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-federate-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.138344 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.138296    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-serving-certs-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.139002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.138907    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-metrics-client-ca\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.139172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.139127    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.141015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.140598    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.141015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.140971    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.141149 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.141072    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-telemeter-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.141149 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.141137    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/479b5a03-a117-4edf-8da2-92f43a614802-federate-client-tls\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.197979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.197950    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qztb\" (UniqueName: \"kubernetes.io/projected/479b5a03-a117-4edf-8da2-92f43a614802-kube-api-access-6qztb\") pod \"telemeter-client-6fc9f56dfc-cxb5q\" (UID: \"479b5a03-a117-4edf-8da2-92f43a614802\") " pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.336959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.336691    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"
Oct 30 21:38:17.789865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:17.789826    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q"]
Oct 30 21:38:17.808159 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:17.808131    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479b5a03_a117_4edf_8da2_92f43a614802.slice/crio-b477d0616291260bd88d6ac3842de7544c20d48596334ce274fca79010ba8513 WatchSource:0}: Error finding container b477d0616291260bd88d6ac3842de7544c20d48596334ce274fca79010ba8513: Status 404 returned error can't find the container with id b477d0616291260bd88d6ac3842de7544c20d48596334ce274fca79010ba8513
Oct 30 21:38:18.108077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.108030    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5dd65c67-2pfcs" event={"ID":"d5c91726-0d2c-40f0-a0a3-ecfe3830def9","Type":"ContainerStarted","Data":"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"}
Oct 30 21:38:18.114629 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.114563    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"0eb444a818771f8bd3c1627761231757e18b90fdbe0b953ef0e63832fc313a1c"}
Oct 30 21:38:18.114784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.114669    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"e79dc983f0933e4216427c72cbb456e6e1d6717b23833b2d2027f10f0f3fe623"}
Oct 30 21:38:18.114784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.114714    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"eea30d9cf91bd68dfdd8d42559422aedfd69b45c55d55214e635b3834fce852e"}
Oct 30 21:38:18.116179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.116110    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q" event={"ID":"479b5a03-a117-4edf-8da2-92f43a614802","Type":"ContainerStarted","Data":"b477d0616291260bd88d6ac3842de7544c20d48596334ce274fca79010ba8513"}
Oct 30 21:38:18.224808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:18.223437    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5c5dd65c67-2pfcs" podStartSLOduration=1.531788205 podStartE2EDuration="5.223422084s" totalImagesPullingTime="3.691633879s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:18.222913503 +0000 UTC m=+21.396237934" watchObservedRunningTime="2026-10-30 21:38:18.223422084 +0000 UTC m=+21.396746509"
Oct 30 21:38:19.123736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:19.123702    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz" event={"ID":"7bb75240-abd5-43e1-aa11-7604abed835d","Type":"ContainerStarted","Data":"4bc46105a4c97dd81ab51766d7494909b847a7b7ea7131e67412d0d85ec8dc95"}
Oct 30 21:38:19.124486 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:19.124297    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:19.130606 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:19.130526    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz"
Oct 30 21:38:19.234518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:19.234461    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-77f4f76594-fnwfz" podStartSLOduration=1.943114997 podStartE2EDuration="4.234445393s" totalImagesPullingTime="2.291330396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:19.233687939 +0000 UTC m=+22.407012369" watchObservedRunningTime="2026-10-30 21:38:19.234445393 +0000 UTC m=+22.407769820"
Oct 30 21:38:20.130173 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.130141    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"8c2c31587942740e48780b460d209ca4d78d204a45b1404d86474b2f09927cd4"}
Oct 30 21:38:20.130663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.130212    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"24462eff61f348088f380ef1d751de2a1d9d5178c9b080536cbe5c4315e1f303"}
Oct 30 21:38:20.130663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.130245    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" event={"ID":"2024b1bc-1853-477d-8ae2-8329bd3dbb8f","Type":"ContainerStarted","Data":"fd82bcd9b5f103bddc111c26c96960435f3bf94fcade60cf598e8c0fdf171411"}
Oct 30 21:38:20.130663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.130396    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:20.134106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134082    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"}
Oct 30 21:38:20.134259 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134150    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"}
Oct 30 21:38:20.134259 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134194    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"}
Oct 30 21:38:20.134259 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134230    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"}
Oct 30 21:38:20.134415 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"}
Oct 30 21:38:20.134415 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.134311    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerStarted","Data":"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"}
Oct 30 21:38:20.136293 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.136264    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s" event={"ID":"33313b40-bb78-46ad-8229-08f46def4870","Type":"ContainerStarted","Data":"cfdfd2250cc42584a3e2fe5cc6334767f25989f65f8b0e9e9e55c246b686db23"}
Oct 30 21:38:20.263858 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.263755    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x" podStartSLOduration=2.7287613090000002 podStartE2EDuration="7.263738789s" totalImagesPullingTime="4.53497748s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:20.263251178 +0000 UTC m=+23.436575605" watchObservedRunningTime="2026-10-30 21:38:20.263738789 +0000 UTC m=+23.437063211"
Oct 30 21:38:20.389447 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.389378    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s" podStartSLOduration=2.138613891 podStartE2EDuration="4.389363588s" totalImagesPullingTime="2.250749697s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:20.388481676 +0000 UTC m=+23.561806157" watchObservedRunningTime="2026-10-30 21:38:20.389363588 +0000 UTC m=+23.562688013"
Oct 30 21:38:20.421556 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.421532    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:38:20.427271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.427247    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464196 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464207    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464227    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464307    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464421    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464521 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464479    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464521 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464502    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464607 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464607 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464555    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464614    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464635    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464650    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464666    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464682 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464680    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464712    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464740    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464847    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464879    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvjh7\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.464979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.464919    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.483357 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.483337    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:38:20.483471 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.483454    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:38:20.483637 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.483623    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:38:20.483813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.483795    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:38:20.484692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.484677    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:38:20.484754 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.484690    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3kccurv99vp74\""
Oct 30 21:38:20.484930 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.484916    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:38:20.484975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.484966    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:38:20.485020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.485004    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:38:20.485020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.485006    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tgwpl\""
Oct 30 21:38:20.485083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.485018    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:38:20.485083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.485023    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:38:20.485083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.485041    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:38:20.532643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.532513    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:38:20.532823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.532642    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:38:20.532917 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.532827    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:38:20.535083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.535064    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:38:20.566050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566055    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566085    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566103    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566123    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566289    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566323    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566360    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566386    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566410    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566436    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566477    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566501    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvjh7\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566533    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566580    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566647    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566678    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566708    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.566945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.566774    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567026    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567070    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567156    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567344 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567333    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567442    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.567798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.567643    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.569288 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.569259    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.570432 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.570401    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.570545 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.570404    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.570678 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.570647    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.570829 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.570806    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.571209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.571130    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.571355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.571333    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.572195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.572169    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.572299 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.572261    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.572850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.572824    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.572983 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.572967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.573382 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.573363    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.573769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.573748    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.577739 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.577698    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.243696992 podStartE2EDuration="7.577687072s" totalImagesPullingTime="4.308323295s" totalInitContainerRuntime="25.666785ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:20.577423812 +0000 UTC m=+23.750748243" watchObservedRunningTime="2026-10-30 21:38:20.577687072 +0000 UTC m=+23.751011498"
Oct 30 21:38:20.634662 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.634629    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvjh7\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\") pod \"prometheus-k8s-0\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.774708 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:20.774671    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:20.806194 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:20.806158    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:38:21.141645 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:21.141606    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q" event={"ID":"479b5a03-a117-4edf-8da2-92f43a614802","Type":"ContainerStarted","Data":"d8971355a2283782cb21d70c75dd63c820f16e3bc4b469a17517411190bb0a1f"}
Oct 30 21:38:21.142156 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:21.141669    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q" event={"ID":"479b5a03-a117-4edf-8da2-92f43a614802","Type":"ContainerStarted","Data":"578531427ff5abbad58e356deced6e90771d562aa33bec5b3693ead519744ada"}
Oct 30 21:38:21.142156 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:21.141690    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q" event={"ID":"479b5a03-a117-4edf-8da2-92f43a614802","Type":"ContainerStarted","Data":"86867eb0c8e1ff60f924988cf2404901fcf6ce14e46ed34320423e431eda2574"}
Oct 30 21:38:21.229951 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:21.229924    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:38:21.231457 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:38:21.231432    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8b5953a_503f_4ef8_936f_d3d4a129a833.slice/crio-b5afac8e9670edeffcc9ce7f310e689c625a79195373e68eabb4b9a63498013e WatchSource:0}: Error finding container b5afac8e9670edeffcc9ce7f310e689c625a79195373e68eabb4b9a63498013e: Status 404 returned error can't find the container with id b5afac8e9670edeffcc9ce7f310e689c625a79195373e68eabb4b9a63498013e
Oct 30 21:38:21.303914 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:21.303866    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6fc9f56dfc-cxb5q" podStartSLOduration=2.846174498 podStartE2EDuration="5.303849695s" totalImagesPullingTime="2.457675197s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:21.30232662 +0000 UTC m=+24.475651049" watchObservedRunningTime="2026-10-30 21:38:21.303849695 +0000 UTC m=+24.477174120"
Oct 30 21:38:22.146703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:22.146670    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478" exitCode=0
Oct 30 21:38:22.147147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:22.146767    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"}
Oct 30 21:38:22.147147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:22.146863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"b5afac8e9670edeffcc9ce7f310e689c625a79195373e68eabb4b9a63498013e"}
Oct 30 21:38:23.501318 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:23.501276    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:23.501954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:23.501351    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:23.506783 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:23.506753    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:24.158007 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.157973    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:38:24.158179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.158017    4882 generic.go:356] "Generic (PLEG): container finished" podID="773ef005-be9c-4f87-b282-c5fad993d143" containerID="8db23c878948cca4fe36a8f0f57ef69a25bf02282875c75091b7bcd846174232" exitCode=1
Oct 30 21:38:24.158179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.158116    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerDied","Data":"8db23c878948cca4fe36a8f0f57ef69a25bf02282875c75091b7bcd846174232"}
Oct 30 21:38:24.158769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.158733    4882 scope.go:122] "RemoveContainer" containerID="8db23c878948cca4fe36a8f0f57ef69a25bf02282875c75091b7bcd846174232"
Oct 30 21:38:24.162543 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.162526    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:38:24.919581 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:24.919548    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-sxl4l"
Oct 30 21:38:25.166830 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:25.166796    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:38:25.167016 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:25.166895    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-7pfdn" event={"ID":"773ef005-be9c-4f87-b282-c5fad993d143","Type":"ContainerStarted","Data":"25b23b4b05823c96675a86fd5b9a4969c34106b08f618afddfca06f28c86313f"}
Oct 30 21:38:25.779604 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:25.779563    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:26.147926 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.147001    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-fc58c9df8-bjk8x"
Oct 30 21:38:26.175695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175658    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"}
Oct 30 21:38:26.175872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175725    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"}
Oct 30 21:38:26.175872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175750    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"}
Oct 30 21:38:26.175872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175778    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"}
Oct 30 21:38:26.175872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175815    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"}
Oct 30 21:38:26.175872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.175857    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerStarted","Data":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"}
Oct 30 21:38:26.483128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:26.483052    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.849682087 podStartE2EDuration="6.483037131s" totalImagesPullingTime="3.617878686s" totalInitContainerRuntime="15.476358ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:38:26.482893616 +0000 UTC m=+29.656218044" watchObservedRunningTime="2026-10-30 21:38:26.483037131 +0000 UTC m=+29.656361569"
Oct 30 21:38:30.774860 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:30.774822    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:31.218122 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:31.218089    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:33.304909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:33.304877    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fdqv5"
Oct 30 21:38:38.870603 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:38.870555    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:38:45.785582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:45.785551    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:45.789532 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:45.789509    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-767dd57b75-mqc9s"
Oct 30 21:38:47.040555 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:47.040513    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:47.060798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:47.060767    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:47.281772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:47.281743    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:38:50.804600 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:50.804543    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:38:58.009513 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009479    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:58.010079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009819    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="alertmanager" containerID="cri-o://088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f" gracePeriod=120
Oct 30 21:38:58.010079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009873    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-metric" containerID="cri-o://cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2" gracePeriod=120
Oct 30 21:38:58.010079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009901    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="prom-label-proxy" containerID="cri-o://e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47" gracePeriod=120
Oct 30 21:38:58.010079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009939    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy" containerID="cri-o://4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305" gracePeriod=120
Oct 30 21:38:58.010079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.009982    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="config-reloader" containerID="cri-o://9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc" gracePeriod=120
Oct 30 21:38:58.010352 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.010071    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-web" containerID="cri-o://401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1" gracePeriod=120
Oct 30 21:38:58.309966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.309874    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47" exitCode=0
Oct 30 21:38:58.309966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.309898    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305" exitCode=0
Oct 30 21:38:58.309966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.309914    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc" exitCode=0
Oct 30 21:38:58.309966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.309921    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f" exitCode=0
Oct 30 21:38:58.309966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.309948    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"}
Oct 30 21:38:58.310278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.310045    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"}
Oct 30 21:38:58.310278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.310085    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"}
Oct 30 21:38:58.310278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:58.310120    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"}
Oct 30 21:38:59.268176 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.268147    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.317728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317694    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2" exitCode=0
Oct 30 21:38:59.317728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317719    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerID="401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1" exitCode=0
Oct 30 21:38:59.317961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317749    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"}
Oct 30 21:38:59.317961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317769    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.317961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317830    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"}
Oct 30 21:38:59.317961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317873    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9","Type":"ContainerDied","Data":"77db7289b6003f430a95936b70983e11bbcd28f6564ea2ac0733719d2a33b6d2"}
Oct 30 21:38:59.317961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.317894    4882 scope.go:122] "RemoveContainer" containerID="e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"
Oct 30 21:38:59.326430 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.326411    4882 scope.go:122] "RemoveContainer" containerID="cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"
Oct 30 21:38:59.335785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.335761    4882 scope.go:122] "RemoveContainer" containerID="4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"
Oct 30 21:38:59.336374 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336351    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336465 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336406    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336465 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336443    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336566 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336473    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336566 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336512    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336566 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336547    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336606    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336675    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336704    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336739    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.336805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336792    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.337059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336832    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.337059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336859    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\") pod \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\" (UID: \"7241e2f5-2bda-4563-b53f-2fb03e9fb4f9\") "
Oct 30 21:38:59.337059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.336942    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:59.337264 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.337172    4882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-main-db\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.337942 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.337877    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:59.338054 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.337943    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:38:59.339446 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.339415    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:38:59.339771 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.339616    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:59.339771 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.339734    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.340043 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.340009    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "kube-api-access-d8sbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:38:59.340338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.340313    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.341038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.341012    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.341479 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.341454    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.342992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.342965    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.345857 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.345184    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.353115 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.353087    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config" pod "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" (UID: "7241e2f5-2bda-4563-b53f-2fb03e9fb4f9"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:38:59.353619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.353601    4882 scope.go:122] "RemoveContainer" containerID="401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"
Oct 30 21:38:59.361523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.361505    4882 scope.go:122] "RemoveContainer" containerID="9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"
Oct 30 21:38:59.369059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.369040    4882 scope.go:122] "RemoveContainer" containerID="088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"
Oct 30 21:38:59.376388 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.376368    4882 scope.go:122] "RemoveContainer" containerID="6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"
Oct 30 21:38:59.383419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.383400    4882 scope.go:122] "RemoveContainer" containerID="e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"
Oct 30 21:38:59.383858 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.383830    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47\": container with ID starting with e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47 not found: ID does not exist" containerID="e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"
Oct 30 21:38:59.383954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.383867    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"} err="failed to get container status \"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47\": rpc error: code = NotFound desc = could not find container \"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47\": container with ID starting with e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47 not found: ID does not exist"
Oct 30 21:38:59.383954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.383904    4882 scope.go:122] "RemoveContainer" containerID="cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"
Oct 30 21:38:59.384192 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.384169    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2\": container with ID starting with cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2 not found: ID does not exist" containerID="cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"
Oct 30 21:38:59.384244 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384199    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"} err="failed to get container status \"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2\": rpc error: code = NotFound desc = could not find container \"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2\": container with ID starting with cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2 not found: ID does not exist"
Oct 30 21:38:59.384244 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384215    4882 scope.go:122] "RemoveContainer" containerID="4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"
Oct 30 21:38:59.384454 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.384439    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305\": container with ID starting with 4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305 not found: ID does not exist" containerID="4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"
Oct 30 21:38:59.384505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384458    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"} err="failed to get container status \"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305\": rpc error: code = NotFound desc = could not find container \"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305\": container with ID starting with 4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305 not found: ID does not exist"
Oct 30 21:38:59.384505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384475    4882 scope.go:122] "RemoveContainer" containerID="401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"
Oct 30 21:38:59.384942 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.384709    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1\": container with ID starting with 401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1 not found: ID does not exist" containerID="401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"
Oct 30 21:38:59.384942 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384727    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"} err="failed to get container status \"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1\": rpc error: code = NotFound desc = could not find container \"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1\": container with ID starting with 401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1 not found: ID does not exist"
Oct 30 21:38:59.384942 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.384738    4882 scope.go:122] "RemoveContainer" containerID="9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"
Oct 30 21:38:59.385069 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.384985    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc\": container with ID starting with 9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc not found: ID does not exist" containerID="9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"
Oct 30 21:38:59.385069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385004    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"} err="failed to get container status \"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc\": rpc error: code = NotFound desc = could not find container \"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc\": container with ID starting with 9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc not found: ID does not exist"
Oct 30 21:38:59.385069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385020    4882 scope.go:122] "RemoveContainer" containerID="088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"
Oct 30 21:38:59.385254 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.385239    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f\": container with ID starting with 088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f not found: ID does not exist" containerID="088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"
Oct 30 21:38:59.385299 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385258    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"} err="failed to get container status \"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f\": rpc error: code = NotFound desc = could not find container \"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f\": container with ID starting with 088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f not found: ID does not exist"
Oct 30 21:38:59.385299 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385270    4882 scope.go:122] "RemoveContainer" containerID="6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"
Oct 30 21:38:59.385497 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:38:59.385483    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7\": container with ID starting with 6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7 not found: ID does not exist" containerID="6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"
Oct 30 21:38:59.385531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385500    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"} err="failed to get container status \"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7\": rpc error: code = NotFound desc = could not find container \"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7\": container with ID starting with 6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7 not found: ID does not exist"
Oct 30 21:38:59.385531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385510    4882 scope.go:122] "RemoveContainer" containerID="e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"
Oct 30 21:38:59.385752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385736    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47"} err="failed to get container status \"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47\": rpc error: code = NotFound desc = could not find container \"e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47\": container with ID starting with e230885f45a622e0bbda93579c34cb5dcc058d6065d1d998e5a610851fc5aa47 not found: ID does not exist"
Oct 30 21:38:59.385802 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385751    4882 scope.go:122] "RemoveContainer" containerID="cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"
Oct 30 21:38:59.385981 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385964    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2"} err="failed to get container status \"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2\": rpc error: code = NotFound desc = could not find container \"cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2\": container with ID starting with cdbefa038fe86bfe8b1425843bebce62e34ada4fe717522559b4b28ba66b62f2 not found: ID does not exist"
Oct 30 21:38:59.386022 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.385983    4882 scope.go:122] "RemoveContainer" containerID="4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"
Oct 30 21:38:59.386202 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386184    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305"} err="failed to get container status \"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305\": rpc error: code = NotFound desc = could not find container \"4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305\": container with ID starting with 4171e6897e158c037cba311e706b7a3cd86d3c2a5f769310dedaa0133b235305 not found: ID does not exist"
Oct 30 21:38:59.386202 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386202    4882 scope.go:122] "RemoveContainer" containerID="401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"
Oct 30 21:38:59.386452 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386435    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1"} err="failed to get container status \"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1\": rpc error: code = NotFound desc = could not find container \"401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1\": container with ID starting with 401b54687a8487c3e99e98a123af8bfa586df7972534c794383482e686034ad1 not found: ID does not exist"
Oct 30 21:38:59.386500 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386451    4882 scope.go:122] "RemoveContainer" containerID="9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"
Oct 30 21:38:59.386691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386670    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc"} err="failed to get container status \"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc\": rpc error: code = NotFound desc = could not find container \"9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc\": container with ID starting with 9e6cda7150f7d7b8c236166665f33c1ffd09e47e7ad2030119eddc8bf6151ccc not found: ID does not exist"
Oct 30 21:38:59.386732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386691    4882 scope.go:122] "RemoveContainer" containerID="088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"
Oct 30 21:38:59.386906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386885    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f"} err="failed to get container status \"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f\": rpc error: code = NotFound desc = could not find container \"088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f\": container with ID starting with 088c086a14cc6c03b4279b366fd0ef0115f9f09f5f44241378d6c457969bc32f not found: ID does not exist"
Oct 30 21:38:59.386953 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.386905    4882 scope.go:122] "RemoveContainer" containerID="6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"
Oct 30 21:38:59.387164 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.387134    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7"} err="failed to get container status \"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7\": rpc error: code = NotFound desc = could not find container \"6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7\": container with ID starting with 6bb1b93f403db0eb0fc48aa624439a16c88a596c6523cc5ba0ecd861cd191ad7 not found: ID does not exist"
Oct 30 21:38:59.437559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437532    4882 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437557    4882 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-cluster-tls-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437570    4882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-main-tls\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437579    4882 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437605    4882 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-metrics-client-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437614    4882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437622    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d8sbs\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-kube-api-access-d8sbs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437631    4882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-tls-assets\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437640    4882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437650    4882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-config-out\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437659    4882 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.437789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.437690    4882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9-web-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:38:59.736284 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.736249    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:59.744838 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.744813    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:59.820218 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820186    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:59.820544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820531    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-metric"
Oct 30 21:38:59.820582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820546    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-metric"
Oct 30 21:38:59.820582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820559    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="init-config-reloader"
Oct 30 21:38:59.820582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820564    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="init-config-reloader"
Oct 30 21:38:59.820582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820574    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="alertmanager"
Oct 30 21:38:59.820582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820579    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="alertmanager"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820628    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="prom-label-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820640    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="prom-label-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820652    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="config-reloader"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820657    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="config-reloader"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820668    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-web"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820672    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-web"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820682    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820686    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820739    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="prom-label-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820745    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="config-reloader"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820751    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-web"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820757    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy"
Oct 30 21:38:59.820761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820762    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="kube-rbac-proxy-metric"
Oct 30 21:38:59.821099 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.820768    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" containerName="alertmanager"
Oct 30 21:38:59.863430 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.863399    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.898577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.898501    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:38:59.942059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942015    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-web-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942056    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942140    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-config-out\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942174    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942193    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942437 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942337    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942437 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942427    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxb9t\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-kube-api-access-bxb9t\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942467    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942490    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942615 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942556    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942610    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.942667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.942636    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:38:59.943575 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943552    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943601    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943600    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943632    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943632    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943632    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Oct 30 21:38:59.943737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.943657    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Oct 30 21:38:59.961725 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.961700    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Oct 30 21:38:59.961847 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.961833    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Oct 30 21:38:59.961972 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:38:59.961937    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fj7sr\""
Oct 30 21:39:00.043883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.043847    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-config-out\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.043883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.043886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.043906    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.043921    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.043937    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxb9t\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-kube-api-access-bxb9t\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044059    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044082    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044116    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044141    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044165    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-web-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044233    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.044692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.044545    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.045444 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.045417    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.046216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.046162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1de6cc72-b70a-438b-940c-f4581274b86b-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.046955 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.046903    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1de6cc72-b70a-438b-940c-f4581274b86b-config-out\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.047171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.047117    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.047282 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.047177    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.047378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.047352    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.047455 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.047433    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.048243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.048220    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.048243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.048231    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.048562 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.048540    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-config-volume\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.048728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.048712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1de6cc72-b70a-438b-940c-f4581274b86b-web-config\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.104986 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.104957    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxb9t\" (UniqueName: \"kubernetes.io/projected/1de6cc72-b70a-438b-940c-f4581274b86b-kube-api-access-bxb9t\") pod \"alertmanager-main-0\" (UID: \"1de6cc72-b70a-438b-940c-f4581274b86b\") " pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.316657 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.316547    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Oct 30 21:39:00.813009 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.812986    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Oct 30 21:39:00.814782 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:39:00.814750    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de6cc72_b70a_438b_940c_f4581274b86b.slice/crio-3f754934e4b84ca2c6c15b3fe5f4e3eade3303bd4d723e28ee0ce146842d5cba WatchSource:0}: Error finding container 3f754934e4b84ca2c6c15b3fe5f4e3eade3303bd4d723e28ee0ce146842d5cba: Status 404 returned error can't find the container with id 3f754934e4b84ca2c6c15b3fe5f4e3eade3303bd4d723e28ee0ce146842d5cba
Oct 30 21:39:00.941043 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:00.941008    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7241e2f5-2bda-4563-b53f-2fb03e9fb4f9" path="/var/lib/kubelet/pods/7241e2f5-2bda-4563-b53f-2fb03e9fb4f9/volumes"
Oct 30 21:39:01.328380 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:01.328346    4882 generic.go:356] "Generic (PLEG): container finished" podID="1de6cc72-b70a-438b-940c-f4581274b86b" containerID="fbbc0f7d03ec7e6fa3d2d7f0346d646912e7d382f52609fd2db0fd212ae7b2ac" exitCode=0
Oct 30 21:39:01.328763 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:01.328425    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerDied","Data":"fbbc0f7d03ec7e6fa3d2d7f0346d646912e7d382f52609fd2db0fd212ae7b2ac"}
Oct 30 21:39:01.328763 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:01.328495    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"3f754934e4b84ca2c6c15b3fe5f4e3eade3303bd4d723e28ee0ce146842d5cba"}
Oct 30 21:39:02.335657 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335621    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"563d0b334b2f13a628b15b17e6df9634ef7c777a30448647c2e7384827a9e253"}
Oct 30 21:39:02.336040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335693    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"fabdc5a4c8277825fbea9cfdbdf9073e8575dd62e469706b659f5bf3366adbb6"}
Oct 30 21:39:02.336040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335720    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"2f721c016c822ecf81c8e901fc78d5fbbffd51140f89151cc8708bd8c36d217c"}
Oct 30 21:39:02.336040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335743    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"33f7b788045809acda883a889670736a7b2cf8810f6510b51324df5b0d3f3ebe"}
Oct 30 21:39:02.336040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335768    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"5f6215fb7f50cb962ac423c44bc7f4b164b6638b5095eca1d91fd8faba2939d5"}
Oct 30 21:39:02.336040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.335789    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1de6cc72-b70a-438b-940c-f4581274b86b","Type":"ContainerStarted","Data":"7d67b1a9c8f6cd5bcd106289bed601a0c6a471d0559ae96f3a1a2a07305538ce"}
Oct 30 21:39:02.465207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:02.465152    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.452303369 podStartE2EDuration="3.465135436s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.832067ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:38:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:02.462901031 +0000 UTC m=+65.636225460" watchObservedRunningTime="2026-10-30 21:39:02.465135436 +0000 UTC m=+65.638459902"
Oct 30 21:39:03.891768 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.891725    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5c5dd65c67-2pfcs" podUID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" containerName="console" containerID="cri-o://4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3" gracePeriod=15
Oct 30 21:39:03.900317 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900291    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:03.900637 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900606    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="prometheus" containerID="cri-o://03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692" gracePeriod=600
Oct 30 21:39:03.900731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900627    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-web" containerID="cri-o://e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75" gracePeriod=600
Oct 30 21:39:03.900731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900648    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="config-reloader" containerID="cri-o://4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8" gracePeriod=600
Oct 30 21:39:03.900731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900674    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="thanos-sidecar" containerID="cri-o://e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9" gracePeriod=600
Oct 30 21:39:03.900731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900724    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy" containerID="cri-o://3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b" gracePeriod=600
Oct 30 21:39:03.900944 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:03.900643    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-thanos" containerID="cri-o://3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804" gracePeriod=600
Oct 30 21:39:04.151285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.151256    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:39:04.183454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183412    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183464    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183605    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183779 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183676    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183779 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183713    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\" (UniqueName: \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183779 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183773    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183929 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183799    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183929 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183860    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\") pod \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\" (UID: \"d5c91726-0d2c-40f0-a0a3-ecfe3830def9\") "
Oct 30 21:39:04.183929 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183912    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.184062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.183939    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.184062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184028    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.184170 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184150    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:04.184275 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184248    4882 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-oauth-serving-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.184275 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184266    4882 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-tmp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.184275 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184275    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-trusted-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.184383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184284    4882 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.184494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.184470    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.186463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.186436    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.186541 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.186496    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "kube-api-access-vsvzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:04.186691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.186671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config" pod "d5c91726-0d2c-40f0-a0a3-ecfe3830def9" (UID: "d5c91726-0d2c-40f0-a0a3-ecfe3830def9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.205577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.205553    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.284817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284784    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284830    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284865    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284926    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284958    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.284983    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285019    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285054    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285085    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285115    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285143    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285184    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285214    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285243    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285229    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285292    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285321    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285365    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285398    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285439    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.285695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285470    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\") pod \"a8b5953a-503f-4ef8-936f-d3d4a129a833\" (UID: \"a8b5953a-503f-4ef8-936f-d3d4a129a833\") "
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285705    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285776    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285795    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vsvzb\" (UniqueName: \"kubernetes.io/projected/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-kube-api-access-vsvzb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285814    4882 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-serving-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285828    4882 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/d5c91726-0d2c-40f0-a0a3-ecfe3830def9-console-oauth-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285843    4882 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.285860    4882 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.286345 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.286223    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.287608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.286924    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.287608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.287282    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.287608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.287454    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.287841 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.287815    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:04.287910 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.287869    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:04.288574 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.288541    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.289386 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.289357    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.289890 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.289865    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.290027 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.289870    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:04.290387 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.290360    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.290510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.290487    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:04.290691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.290667    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.290933 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.290909    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.291302 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.291280    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "kube-api-access-cvjh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:04.291581 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.291563    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.291903 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.291885    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.300209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.300183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config" pod "a8b5953a-503f-4ef8-936f-d3d4a129a833" (UID: "a8b5953a-503f-4ef8-936f-d3d4a129a833"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:39:04.344659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.344619    4882 generic.go:356] "Generic (PLEG): container finished" podID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" containerID="4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3" exitCode=0
Oct 30 21:39:04.344659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.344661    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5c5dd65c67-2pfcs"
Oct 30 21:39:04.344874 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.344689    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5dd65c67-2pfcs" event={"ID":"d5c91726-0d2c-40f0-a0a3-ecfe3830def9","Type":"ContainerDied","Data":"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"}
Oct 30 21:39:04.344874 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.344738    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5c5dd65c67-2pfcs" event={"ID":"d5c91726-0d2c-40f0-a0a3-ecfe3830def9","Type":"ContainerDied","Data":"41b919e1de8511ba47c4104bedf2aa1c350915f477f5c87409ab86798bed5012"}
Oct 30 21:39:04.344874 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.344759    4882 scope.go:122] "RemoveContainer" containerID="4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"
Oct 30 21:39:04.348062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348028    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804" exitCode=0
Oct 30 21:39:04.348062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348060    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b" exitCode=0
Oct 30 21:39:04.348062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348069    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75" exitCode=0
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348075    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9" exitCode=0
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348081    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8" exitCode=0
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348087    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692" exitCode=0
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348101    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348067    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348168    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348197    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348230    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348253    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"}
Oct 30 21:39:04.348311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.348307    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a8b5953a-503f-4ef8-936f-d3d4a129a833","Type":"ContainerDied","Data":"b5afac8e9670edeffcc9ce7f310e689c625a79195373e68eabb4b9a63498013e"}
Oct 30 21:39:04.354060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.354034    4882 scope.go:122] "RemoveContainer" containerID="4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"
Oct 30 21:39:04.354395 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.354378    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3\": container with ID starting with 4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3 not found: ID does not exist" containerID="4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"
Oct 30 21:39:04.354447 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.354405    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3"} err="failed to get container status \"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3\": rpc error: code = NotFound desc = could not find container \"4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3\": container with ID starting with 4cef90eb286ae60bea398c57ac85cee16d1cbf971688388bd3e3a20c1a08bfd3 not found: ID does not exist"
Oct 30 21:39:04.354447 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.354422    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.362046 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.362028    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.368953 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.368935    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.375879 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.375863    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.382576 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.382556    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.386881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386853    4882 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-web-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.386881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386879    4882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386891    4882 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-kube-rbac-proxy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386899    4882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386912    4882 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386922    4882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386931    4882 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-grpc-tls\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386940    4882 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386949    4882 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-tls-assets\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386957    4882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-db\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386965    4882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386976    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvjh7\" (UniqueName: \"kubernetes.io/projected/a8b5953a-503f-4ef8-936f-d3d4a129a833-kube-api-access-cvjh7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386986    4882 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-configmap-metrics-client-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.386995    4882 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a8b5953a-503f-4ef8-936f-d3d4a129a833-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.387004    4882 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-metrics-client-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.387012    4882 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-config\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.387020    4882 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a8b5953a-503f-4ef8-936f-d3d4a129a833-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.387424 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.387029    4882 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a8b5953a-503f-4ef8-936f-d3d4a129a833-config-out\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:04.389846 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.389826    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.397632 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.397613    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.404975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.404952    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.405308 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.405285    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.405360 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405318    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.405360 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405338    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.405624 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.405605    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.405681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405630    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.405681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405646    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.405892 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.405874    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.405985 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405901    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.405985 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.405927    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.406187 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.406167    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.406285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406192    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.406285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406208    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.406549 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.406523    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.406620 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406549    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.406620 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406563    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.406951 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.406929    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.407044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406967    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.407044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.406985    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.407308 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:04.407292    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.407367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.407325    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.407367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.407344    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.407652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.407576    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.407706 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.407653    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.408087 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.408056    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.408200 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.408085    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.408854 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.408406    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.408854 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.408431    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.409034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.408999    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.409034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.409024    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.409319 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.409291    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.409319 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.409319    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.409753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.409657    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.409753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.409682    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.410065 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410003    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.410065 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410028    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.410609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410564    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.410710 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410612    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.410890 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410871    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.410952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.410891    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.411285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411263    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.411285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411284    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.411578 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411555    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.411578 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411578    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.411954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411934    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.411954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.411954    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.412210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412193    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.412276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412210    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.412506 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412474    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.412506 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412502    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.412807 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412789    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.412869 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.412807    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.413074 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413050    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.413074 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413074    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.413393 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413356    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.413393 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413393    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.413680 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413650    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.413680 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413674    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.413984 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413936    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.413984 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.413980    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.414247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414219    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.414247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414241    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.414499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414471    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.414499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414490    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.414724 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414705    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.414786 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414725    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.414925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414906    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.414977 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.414926    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.415150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415127    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.415150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415148    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.415420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415403    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.415420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415419    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.415714 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415691    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.415790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415716    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.416013 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.415993    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.416078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416011    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.416301 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416285    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.416360 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416301    4882 scope.go:122] "RemoveContainer" containerID="3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"
Oct 30 21:39:04.416548 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416527    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804"} err="failed to get container status \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": rpc error: code = NotFound desc = could not find container \"3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804\": container with ID starting with 3c38b29cac81619b229cbe0664e3b555fa67593b70a2c58a4179e5ac014ac804 not found: ID does not exist"
Oct 30 21:39:04.416548 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416546    4882 scope.go:122] "RemoveContainer" containerID="3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"
Oct 30 21:39:04.416812 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416793    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b"} err="failed to get container status \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": rpc error: code = NotFound desc = could not find container \"3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b\": container with ID starting with 3c30fcba432a6c402dc01f771ec875c41e31bd686d19dbc36ef8a4c967423b6b not found: ID does not exist"
Oct 30 21:39:04.416812 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.416811    4882 scope.go:122] "RemoveContainer" containerID="e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"
Oct 30 21:39:04.417060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417034    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75"} err="failed to get container status \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": rpc error: code = NotFound desc = could not find container \"e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75\": container with ID starting with e071f4e94c6be786e6d8f599fa22062d7b8aa59488c04343adcfd3695c18fb75 not found: ID does not exist"
Oct 30 21:39:04.417110 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417062    4882 scope.go:122] "RemoveContainer" containerID="e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"
Oct 30 21:39:04.417302 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417276    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9"} err="failed to get container status \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": rpc error: code = NotFound desc = could not find container \"e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9\": container with ID starting with e2100e0792081c0f6011e1613bdddbfd44a50fbc2023b292ad87dbba917cd1d9 not found: ID does not exist"
Oct 30 21:39:04.417302 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417295    4882 scope.go:122] "RemoveContainer" containerID="4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"
Oct 30 21:39:04.417480 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417466    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8"} err="failed to get container status \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": rpc error: code = NotFound desc = could not find container \"4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8\": container with ID starting with 4da4c788251312c23feb8fac1c2794e477455a2ea4651713942599986cf963c8 not found: ID does not exist"
Oct 30 21:39:04.417480 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417479    4882 scope.go:122] "RemoveContainer" containerID="03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"
Oct 30 21:39:04.417683 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417663    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692"} err="failed to get container status \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": rpc error: code = NotFound desc = could not find container \"03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692\": container with ID starting with 03de11c3b0dba49055dcc2b2bf82dd1b513c7420c445b47873219bdd22ea2692 not found: ID does not exist"
Oct 30 21:39:04.417727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417683    4882 scope.go:122] "RemoveContainer" containerID="45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"
Oct 30 21:39:04.417922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.417901    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478"} err="failed to get container status \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": rpc error: code = NotFound desc = could not find container \"45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478\": container with ID starting with 45d92442755add21e29e330045f0e1f886c84013f8dcd05c5d1e4107e322e478 not found: ID does not exist"
Oct 30 21:39:04.515685 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.515648    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:39:04.524499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.524468    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5c5dd65c67-2pfcs"]
Oct 30 21:39:04.635711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.635678    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:04.649510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.649437    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:04.690987 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.690955    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:04.691285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691270    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="init-config-reloader"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691286    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="init-config-reloader"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691297    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="thanos-sidecar"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691302    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="thanos-sidecar"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691310    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-web"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691314    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-web"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691320    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy"
Oct 30 21:39:04.691324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691324    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691332    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-thanos"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691337    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-thanos"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691348    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="prometheus"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691352    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="prometheus"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691363    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" containerName="console"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691368    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" containerName="console"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691376    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="config-reloader"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691380    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="config-reloader"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691434    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" containerName="console"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691441    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="config-reloader"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691450    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691457    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="thanos-sidecar"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691461    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-web"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691467    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="kube-rbac-proxy-thanos"
Oct 30 21:39:04.691546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.691473    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" containerName="prometheus"
Oct 30 21:39:04.697042 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.697018    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.755036 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755004    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Oct 30 21:39:04.755206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755098    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Oct 30 21:39:04.755206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755134    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Oct 30 21:39:04.755206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755145    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Oct 30 21:39:04.755206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755148    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Oct 30 21:39:04.755206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755177    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Oct 30 21:39:04.755387 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755256    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Oct 30 21:39:04.755387 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755370    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Oct 30 21:39:04.755467 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755395    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-tgwpl\""
Oct 30 21:39:04.755467 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755396    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Oct 30 21:39:04.755467 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755432    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Oct 30 21:39:04.755572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.755489    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3kccurv99vp74\""
Oct 30 21:39:04.756017 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.756002    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Oct 30 21:39:04.789767 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789735    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.789767 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789789    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789834    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789894    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789909    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.789972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-config-out\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790127    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790243    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790347 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790268    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790347 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790316    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790372    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790394    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-web-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790503 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790412    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790503 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790427    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790503 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790469    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790528    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-696pd\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-kube-api-access-696pd\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.790652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.790581    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.802155 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.802126    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Oct 30 21:39:04.802688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.802670    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Oct 30 21:39:04.802801 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.802786    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Oct 30 21:39:04.804338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.804315    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:04.891107 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891069    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891107 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891108    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-696pd\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-kube-api-access-696pd\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891138    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891179    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891204    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891231    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891260    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891294    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891315    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891377    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-config-out\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891436    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891467    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891500    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891526    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891550    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891625    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891654    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-web-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891682    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.891769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.891707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.892570 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.892025    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.892570 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.892310    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.893037 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.892777    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.894613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.894327    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.894613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.894365    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.894613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.894509    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.894815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.894796    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.894955 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.894933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895038    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895302    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895326    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895623    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/697441e8-bc22-474a-9bc0-d8506d0b40da-config-out\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895679    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895813    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.895990 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.895972    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.896815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.896795    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-web-config\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.897035 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.897015    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.897470 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.897450    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/697441e8-bc22-474a-9bc0-d8506d0b40da-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.898250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.898233    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/697441e8-bc22-474a-9bc0-d8506d0b40da-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:04.941151 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.941119    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8b5953a-503f-4ef8-936f-d3d4a129a833" path="/var/lib/kubelet/pods/a8b5953a-503f-4ef8-936f-d3d4a129a833/volumes"
Oct 30 21:39:04.941809 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.941795    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5c91726-0d2c-40f0-a0a3-ecfe3830def9" path="/var/lib/kubelet/pods/d5c91726-0d2c-40f0-a0a3-ecfe3830def9/volumes"
Oct 30 21:39:04.957656 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:04.957630    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-696pd\" (UniqueName: \"kubernetes.io/projected/697441e8-bc22-474a-9bc0-d8506d0b40da-kube-api-access-696pd\") pod \"prometheus-k8s-0\" (UID: \"697441e8-bc22-474a-9bc0-d8506d0b40da\") " pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:05.028769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:05.028724    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:05.458217 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:05.458186    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Oct 30 21:39:05.460874 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:39:05.460843    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod697441e8_bc22_474a_9bc0_d8506d0b40da.slice/crio-d0b8e9788e8bcbb1accdbc410cd1db687b1a7ad0e334dd2f273d75efeffe7d00 WatchSource:0}: Error finding container d0b8e9788e8bcbb1accdbc410cd1db687b1a7ad0e334dd2f273d75efeffe7d00: Status 404 returned error can't find the container with id d0b8e9788e8bcbb1accdbc410cd1db687b1a7ad0e334dd2f273d75efeffe7d00
Oct 30 21:39:06.358811 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:06.358772    4882 generic.go:356] "Generic (PLEG): container finished" podID="697441e8-bc22-474a-9bc0-d8506d0b40da" containerID="67e80f9052bbf3dce564b51b41b8c81c5cf4295d012e084d7a33ec92e46373c4" exitCode=0
Oct 30 21:39:06.359192 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:06.358860    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerDied","Data":"67e80f9052bbf3dce564b51b41b8c81c5cf4295d012e084d7a33ec92e46373c4"}
Oct 30 21:39:06.359192 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:06.358973    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"d0b8e9788e8bcbb1accdbc410cd1db687b1a7ad0e334dd2f273d75efeffe7d00"}
Oct 30 21:39:07.366655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"d87cfe78b0b3e1640274821e207a89b326217049a0a426d5a435a7124aad56f4"}
Oct 30 21:39:07.367034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366701    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"fbd835e216d64a280d732f6407faa88c2796a118887911fc559e33bd7e891837"}
Oct 30 21:39:07.367034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366746    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"4fcab78c348efd69f71c6090e8c20fd29bd972acb60c8f36911ccd40344019d2"}
Oct 30 21:39:07.367034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366784    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"67f3e0c111eede7424c32feb698483137f380ee8815536d238e4575ef0cbde26"}
Oct 30 21:39:07.367034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366819    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"33a24a6aeda41c15ad4fda1a9bc26ff70203e11ca74014dfccb9cb1227d836af"}
Oct 30 21:39:07.367034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.366854    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"697441e8-bc22-474a-9bc0-d8506d0b40da","Type":"ContainerStarted","Data":"c7b5cbfd06ff38e2408248252dc558c297b44dadb4db9bb33d291f7484b58f8a"}
Oct 30 21:39:07.503004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:07.502940    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.486441695 podStartE2EDuration="3.502922971s" totalImagesPullingTime="0s" totalInitContainerRuntime="16.481276ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:39:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:39:07.500271547 +0000 UTC m=+70.673595973" watchObservedRunningTime="2026-10-30 21:39:07.502922971 +0000 UTC m=+70.676247417"
Oct 30 21:39:08.371667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:08.371626    4882 generic.go:356] "Generic (PLEG): container finished" podID="f3de0e99-5c49-4347-878a-58dbd3fd404b" containerID="9ce770983b879bc3f104f1e576b97c68ab4337aa2a8da56dc42cdba4bcc3798c" exitCode=0
Oct 30 21:39:08.372171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:08.371705    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sw5w-75z9c" event={"ID":"f3de0e99-5c49-4347-878a-58dbd3fd404b","Type":"ContainerDied","Data":"9ce770983b879bc3f104f1e576b97c68ab4337aa2a8da56dc42cdba4bcc3798c"}
Oct 30 21:39:09.509655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.509566    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:39:09.523379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.523355    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:39:09.632018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.631929    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") pod \"f3de0e99-5c49-4347-878a-58dbd3fd404b\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") "
Oct 30 21:39:09.632018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.631972    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") pod \"f3de0e99-5c49-4347-878a-58dbd3fd404b\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") "
Oct 30 21:39:09.632018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.632001    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") pod \"f3de0e99-5c49-4347-878a-58dbd3fd404b\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") "
Oct 30 21:39:09.632314 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.632034    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") pod \"f3de0e99-5c49-4347-878a-58dbd3fd404b\" (UID: \"f3de0e99-5c49-4347-878a-58dbd3fd404b\") "
Oct 30 21:39:09.632379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.632325    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path" pod "f3de0e99-5c49-4347-878a-58dbd3fd404b" (UID: "f3de0e99-5c49-4347-878a-58dbd3fd404b"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:39:09.632379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.632349    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle" pod "f3de0e99-5c49-4347-878a-58dbd3fd404b" (UID: "f3de0e99-5c49-4347-878a-58dbd3fd404b"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:09.632467 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.632400    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle" pod "f3de0e99-5c49-4347-878a-58dbd3fd404b" (UID: "f3de0e99-5c49-4347-878a-58dbd3fd404b"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:39:09.634403 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.634380    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq" pod "f3de0e99-5c49-4347-878a-58dbd3fd404b" (UID: "f3de0e99-5c49-4347-878a-58dbd3fd404b"). InnerVolumeSpecName "kube-api-access-zmrfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:39:09.733520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.733481    4882 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-trusted-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:09.733520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.733513    4882 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/f3de0e99-5c49-4347-878a-58dbd3fd404b-archives-path\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:09.733520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.733524    4882 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f3de0e99-5c49-4347-878a-58dbd3fd404b-service-ca-bundle\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:09.733792 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:09.733533    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmrfq\" (UniqueName: \"kubernetes.io/projected/f3de0e99-5c49-4347-878a-58dbd3fd404b-kube-api-access-zmrfq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:39:10.029396 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.029306    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:39:10.380330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.380301    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-7sw5w-75z9c"
Oct 30 21:39:10.380330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.380310    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-7sw5w-75z9c" event={"ID":"f3de0e99-5c49-4347-878a-58dbd3fd404b","Type":"ContainerDied","Data":"ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa"}
Oct 30 21:39:10.380539 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.380358    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce42b28ba6f668d20f1b876f3aa3945f8f0bdb9a5050a4292cc2d48c70cbe8aa"
Oct 30 21:39:10.635223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.635131    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:39:10.646523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.646491    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-insights/periodic-gathering-7sw5w-75z9c"]
Oct 30 21:39:10.940647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:39:10.940563    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3de0e99-5c49-4347-878a-58dbd3fd404b" path="/var/lib/kubelet/pods/f3de0e99-5c49-4347-878a-58dbd3fd404b/volumes"
Oct 30 21:39:20.804090 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:20.804052    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:39:50.804958 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:39:50.804878    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:40:05.029857 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:40:05.029814    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:40:05.045544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:40:05.045512    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:40:05.605946 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:40:05.605916    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Oct 30 21:40:20.805725 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:40:20.805687    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:40:50.804541 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:40:50.804503    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:41:20.809296 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:41:20.809256    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:41:50.805113 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:41:50.805076    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:41:52.033714 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.033679    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"]
Oct 30 21:41:52.034125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.034045    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f3de0e99-5c49-4347-878a-58dbd3fd404b" containerName="insights-gathering"
Oct 30 21:41:52.034125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.034060    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3de0e99-5c49-4347-878a-58dbd3fd404b" containerName="insights-gathering"
Oct 30 21:41:52.034220 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.034145    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3de0e99-5c49-4347-878a-58dbd3fd404b" containerName="insights-gathering"
Oct 30 21:41:52.037092 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.037076    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.066441 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.066399    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lfjd\" (UniqueName: \"kubernetes.io/projected/58936c80-0e8c-4a9a-ac87-5db90c0c9993-kube-api-access-2lfjd\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.066639 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.066541    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58936c80-0e8c-4a9a-ac87-5db90c0c9993-tmp\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.134535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.134497    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Oct 30 21:41:52.134535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.134497    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Oct 30 21:41:52.134774 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.134570    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-operator-dockercfg-gh79d\""
Oct 30 21:41:52.136404 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.136374    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"]
Oct 30 21:41:52.167153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.167115    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lfjd\" (UniqueName: \"kubernetes.io/projected/58936c80-0e8c-4a9a-ac87-5db90c0c9993-kube-api-access-2lfjd\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.167319 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.167200    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58936c80-0e8c-4a9a-ac87-5db90c0c9993-tmp\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.167568 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.167548    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/58936c80-0e8c-4a9a-ac87-5db90c0c9993-tmp\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.229033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.228990    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lfjd\" (UniqueName: \"kubernetes.io/projected/58936c80-0e8c-4a9a-ac87-5db90c0c9993-kube-api-access-2lfjd\") pod \"jobset-operator-674cd59dc5-wngtx\" (UID: \"58936c80-0e8c-4a9a-ac87-5db90c0c9993\") " pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.367200 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.367167    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"
Oct 30 21:41:52.756236 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:52.756185    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx"]
Oct 30 21:41:52.759074 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:41:52.759045    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod58936c80_0e8c_4a9a_ac87_5db90c0c9993.slice/crio-300feaad86509a62f75992628766c54df6b3c467335bd48cc2c2c0c31030f8bc WatchSource:0}: Error finding container 300feaad86509a62f75992628766c54df6b3c467335bd48cc2c2c0c31030f8bc: Status 404 returned error can't find the container with id 300feaad86509a62f75992628766c54df6b3c467335bd48cc2c2c0c31030f8bc
Oct 30 21:41:53.004804 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:53.004768    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx" event={"ID":"58936c80-0e8c-4a9a-ac87-5db90c0c9993","Type":"ContainerStarted","Data":"300feaad86509a62f75992628766c54df6b3c467335bd48cc2c2c0c31030f8bc"}
Oct 30 21:41:55.846063 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.846024    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"]
Oct 30 21:41:55.856228 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.856193    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:55.892674 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.892635    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmfmc\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-kube-api-access-jmfmc\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:55.892868 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.892693    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:55.907372 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.907312    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"]
Oct 30 21:41:55.952155 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.952117    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Oct 30 21:41:55.952155 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.952149    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Oct 30 21:41:55.954410 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.954396    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-webhook-dockercfg-zbngx\""
Oct 30 21:41:55.993559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.993517    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:55.993760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:55.993659    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmfmc\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-kube-api-access-jmfmc\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:56.053073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:56.053033    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmfmc\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-kube-api-access-jmfmc\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:56.053537 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:56.053514    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/35f59f7a-007c-490c-9861-e33de159492d-bound-sa-token\") pod \"cert-manager-webhook-5dbbbf75cc-zkp79\" (UID: \"35f59f7a-007c-490c-9861-e33de159492d\") " pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:56.172556 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:56.172470    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:41:56.545397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:56.545369    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"]
Oct 30 21:41:56.552766 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:41:56.552733    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod35f59f7a_007c_490c_9861_e33de159492d.slice/crio-8a9f441a0804617a8eac3edcee98c5984574a6a6de3ef30fdd44da30d4f9db89 WatchSource:0}: Error finding container 8a9f441a0804617a8eac3edcee98c5984574a6a6de3ef30fdd44da30d4f9db89: Status 404 returned error can't find the container with id 8a9f441a0804617a8eac3edcee98c5984574a6a6de3ef30fdd44da30d4f9db89
Oct 30 21:41:57.023572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:57.023542    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79" event={"ID":"35f59f7a-007c-490c-9861-e33de159492d","Type":"ContainerStarted","Data":"8a9f441a0804617a8eac3edcee98c5984574a6a6de3ef30fdd44da30d4f9db89"}
Oct 30 21:41:57.025086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:57.025058    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx" event={"ID":"58936c80-0e8c-4a9a-ac87-5db90c0c9993","Type":"ContainerStarted","Data":"83a04366fabec94390da608d3b41a3249c8ff4bd3cd2e5eaf37fb7cb528470d2"}
Oct 30 21:41:57.137247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:41:57.137190    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-operator-674cd59dc5-wngtx" podStartSLOduration=2.375841883 podStartE2EDuration="6.137176303s" totalImagesPullingTime="3.76133442s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:41:57.134778684 +0000 UTC m=+240.308103111" watchObservedRunningTime="2026-10-30 21:41:57.137176303 +0000 UTC m=+240.310500727"
Oct 30 21:42:01.050345 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:01.050305    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79" event={"ID":"35f59f7a-007c-490c-9861-e33de159492d","Type":"ContainerStarted","Data":"3bb13991202038685d1600378d13a5b054bb256b288867a40c2b44a739739e8d"}
Oct 30 21:42:01.050768 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:01.050447    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:42:01.168824 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:01.168764    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79" podStartSLOduration=1.771570235 podStartE2EDuration="6.168748895s" totalImagesPullingTime="4.39717866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:41:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:01.167826277 +0000 UTC m=+244.341150701" watchObservedRunningTime="2026-10-30 21:42:01.168748895 +0000 UTC m=+244.342073321"
Oct 30 21:42:06.123992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.123959    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-qwk6v"]
Oct 30 21:42:06.127471 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.127450    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.183503 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.183467    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-bound-sa-token\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.183679 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.183601    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q77fc\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-kube-api-access-q77fc\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.224171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.224134    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-lcpcf\""
Oct 30 21:42:06.225735 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.225710    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qwk6v"]
Oct 30 21:42:06.284893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.284854    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-bound-sa-token\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.285073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.284918    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q77fc\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-kube-api-access-q77fc\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.342706 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.342673    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-bound-sa-token\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.342900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.342880    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q77fc\" (UniqueName: \"kubernetes.io/projected/2de94d35-11c7-46bc-9d21-82593d349e0b-kube-api-access-q77fc\") pod \"cert-manager-597b95cb84-qwk6v\" (UID: \"2de94d35-11c7-46bc-9d21-82593d349e0b\") " pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.438358 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.438260    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-qwk6v"
Oct 30 21:42:06.806295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:06.806269    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-qwk6v"]
Oct 30 21:42:06.808520 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:42:06.808483    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2de94d35_11c7_46bc_9d21_82593d349e0b.slice/crio-b1b328c4d31b1b936cd67e14c5da81e287347e41f77c4d9e5122175df027eb9a WatchSource:0}: Error finding container b1b328c4d31b1b936cd67e14c5da81e287347e41f77c4d9e5122175df027eb9a: Status 404 returned error can't find the container with id b1b328c4d31b1b936cd67e14c5da81e287347e41f77c4d9e5122175df027eb9a
Oct 30 21:42:07.056742 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:07.056659    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="cert-manager/cert-manager-webhook-5dbbbf75cc-zkp79"
Oct 30 21:42:07.073807 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:07.073772    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qwk6v" event={"ID":"2de94d35-11c7-46bc-9d21-82593d349e0b","Type":"ContainerStarted","Data":"dd69d878b11e41cf41293ed4cc9fb7fe6de3a6b2cc9c9bfde7ab0d79edfe8eab"}
Oct 30 21:42:07.073979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:07.073819    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-qwk6v" event={"ID":"2de94d35-11c7-46bc-9d21-82593d349e0b","Type":"ContainerStarted","Data":"b1b328c4d31b1b936cd67e14c5da81e287347e41f77c4d9e5122175df027eb9a"}
Oct 30 21:42:07.278283 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:07.278216    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-qwk6v" podStartSLOduration=1.278197319 podStartE2EDuration="1.278197319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:07.277708347 +0000 UTC m=+250.451032776" watchObservedRunningTime="2026-10-30 21:42:07.278197319 +0000 UTC m=+250.451521747"
Oct 30 21:42:12.963506 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:12.963473    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"]
Oct 30 21:42:12.967223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:12.967200    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.046822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.046786    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftlvh\" (UniqueName: \"kubernetes.io/projected/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-kube-api-access-ftlvh\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.047018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.046836    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.047018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.046931    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.047125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.047053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.047125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.047113    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.047207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.047140    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.048420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.048400    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nkddb\""
Oct 30 21:42:13.048420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.048410    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Oct 30 21:42:13.048543 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.048479    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Oct 30 21:42:13.049356 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.049328    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Oct 30 21:42:13.049356 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.049330    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Oct 30 21:42:13.049468 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.049407    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Oct 30 21:42:13.098716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.098685    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"]
Oct 30 21:42:13.147702 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147664    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.147899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147708    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.147899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.147899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147764    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftlvh\" (UniqueName: \"kubernetes.io/projected/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-kube-api-access-ftlvh\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.147899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147799    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.147899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.147823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.148173 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.148146    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-manifests\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.150195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.150157    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-tmp\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.150431 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.150411    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.150498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.150434    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-metrics-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.150498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.150459    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-webhook-cert\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.209256 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.209216    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftlvh\" (UniqueName: \"kubernetes.io/projected/921a2af1-bf9b-412c-a4c7-9299e2b6d60c-kube-api-access-ftlvh\") pod \"opendatahub-operator-controller-manager-84bc87c977-6fxdh\" (UID: \"921a2af1-bf9b-412c-a4c7-9299e2b6d60c\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.282232 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.282134    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:13.765464 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:13.765437    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"]
Oct 30 21:42:13.767527 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:42:13.767499    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod921a2af1_bf9b_412c_a4c7_9299e2b6d60c.slice/crio-c6e260cc0b1e2dbcc353c0a701c509ca680e8a7a69696b99991ac9c935a9ec01 WatchSource:0}: Error finding container c6e260cc0b1e2dbcc353c0a701c509ca680e8a7a69696b99991ac9c935a9ec01: Status 404 returned error can't find the container with id c6e260cc0b1e2dbcc353c0a701c509ca680e8a7a69696b99991ac9c935a9ec01
Oct 30 21:42:14.105912 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:14.105855    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh" event={"ID":"921a2af1-bf9b-412c-a4c7-9299e2b6d60c","Type":"ContainerStarted","Data":"c6e260cc0b1e2dbcc353c0a701c509ca680e8a7a69696b99991ac9c935a9ec01"}
Oct 30 21:42:18.126255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:18.126218    4882 generic.go:356] "Generic (PLEG): container finished" podID="921a2af1-bf9b-412c-a4c7-9299e2b6d60c" containerID="e09c5124aea5f68ae8ed3ec8984884c14ecd704dd9a32ff3ef43a8cecc4cb98c" exitCode=0
Oct 30 21:42:18.126721 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:18.126303    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh" event={"ID":"921a2af1-bf9b-412c-a4c7-9299e2b6d60c","Type":"ContainerDied","Data":"e09c5124aea5f68ae8ed3ec8984884c14ecd704dd9a32ff3ef43a8cecc4cb98c"}
Oct 30 21:42:19.133040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:19.133004    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh" event={"ID":"921a2af1-bf9b-412c-a4c7-9299e2b6d60c","Type":"ContainerStarted","Data":"85a298ffda6700e16a328654929e0ee95dfb5e9e53471cc568426e61614daa8b"}
Oct 30 21:42:19.133481 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:19.133149    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:19.318237 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:19.318130    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh" podStartSLOduration=2.273729383 podStartE2EDuration="7.31811406s" totalImagesPullingTime="4.496103957s" totalInitContainerRuntime="548.28072ms" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:19.317109012 +0000 UTC m=+262.490433439" watchObservedRunningTime="2026-10-30 21:42:19.31811406 +0000 UTC m=+262.491438487"
Oct 30 21:42:20.805854 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:42:20.805801    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:42:29.858822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.858783    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"]
Oct 30 21:42:29.886138 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.886101    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:29.916756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.916723    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"]
Oct 30 21:42:29.968266 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.968230    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Oct 30 21:42:29.968453 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.968304    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-wzb94\""
Oct 30 21:42:29.968497 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.968465    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Oct 30 21:42:29.968497 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.968465    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Oct 30 21:42:29.982065 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:29.982037    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Oct 30 21:42:30.011923 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.011892    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8af1fe-acc2-41f4-9146-c47b97930a8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.012083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.011964    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p498\" (UniqueName: \"kubernetes.io/projected/8c8af1fe-acc2-41f4-9146-c47b97930a8a-kube-api-access-5p498\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.012123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.012085    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8af1fe-acc2-41f4-9146-c47b97930a8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.113648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.113548    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p498\" (UniqueName: \"kubernetes.io/projected/8c8af1fe-acc2-41f4-9146-c47b97930a8a-kube-api-access-5p498\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.113808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.113657    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8af1fe-acc2-41f4-9146-c47b97930a8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.113808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.113699    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8af1fe-acc2-41f4-9146-c47b97930a8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.125837 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.114304    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/8c8af1fe-acc2-41f4-9146-c47b97930a8a-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.125837 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.116132    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/8c8af1fe-acc2-41f4-9146-c47b97930a8a-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.140059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.140026    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-84bc87c977-6fxdh"
Oct 30 21:42:30.171572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.171539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p498\" (UniqueName: \"kubernetes.io/projected/8c8af1fe-acc2-41f4-9146-c47b97930a8a-kube-api-access-5p498\") pod \"kuadrant-console-plugin-66c4fb575d-xfkcg\" (UID: \"8c8af1fe-acc2-41f4-9146-c47b97930a8a\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.230100 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.230068    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"
Oct 30 21:42:30.613616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:30.613546    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg"]
Oct 30 21:42:30.621032 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:42:30.621000    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c8af1fe_acc2_41f4_9146_c47b97930a8a.slice/crio-057206838d3f666956bffec641792332956f07b96a0153c039355226eb1fd39d WatchSource:0}: Error finding container 057206838d3f666956bffec641792332956f07b96a0153c039355226eb1fd39d: Status 404 returned error can't find the container with id 057206838d3f666956bffec641792332956f07b96a0153c039355226eb1fd39d
Oct 30 21:42:31.184735 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:31.184700    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg" event={"ID":"8c8af1fe-acc2-41f4-9146-c47b97930a8a","Type":"ContainerStarted","Data":"057206838d3f666956bffec641792332956f07b96a0153c039355226eb1fd39d"}
Oct 30 21:42:38.219790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:38.219750    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg" event={"ID":"8c8af1fe-acc2-41f4-9146-c47b97930a8a","Type":"ContainerStarted","Data":"80dd5388da0a6086219c6f25899db7823044d6c1091eeeffd4df2745184b2a1f"}
Oct 30 21:42:38.329677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:38.329623    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-xfkcg" podStartSLOduration=2.167601792 podStartE2EDuration="9.329608206s" totalImagesPullingTime="7.162006414s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:42:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:42:38.32856469 +0000 UTC m=+281.501889115" watchObservedRunningTime="2026-10-30 21:42:38.329608206 +0000 UTC m=+281.502932629"
Oct 30 21:42:50.805302 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:42:50.805260    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:42:56.952996 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:56.952960    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:42:56.953490 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:42:56.953089    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:43:06.618249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.618213    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"]
Oct 30 21:43:06.623043 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.623020    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.673815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.673781    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Oct 30 21:43:06.674049 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.674023    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Oct 30 21:43:06.722501 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.722468    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"]
Oct 30 21:43:06.742329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.742284    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Oct 30 21:43:06.742329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.742323    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Oct 30 21:43:06.742577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.742391    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-rmm79\""
Oct 30 21:43:06.757258 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.757226    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tls-certs\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.757419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.757349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tmp\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.757419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.757408    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5x7\" (UniqueName: \"kubernetes.io/projected/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-kube-api-access-jd5x7\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.858128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.858090    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tmp\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.858300 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.858141    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5x7\" (UniqueName: \"kubernetes.io/projected/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-kube-api-access-jd5x7\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.858300 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.858261    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tls-certs\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.860424 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.860393    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tmp\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.860649 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.860631    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-tls-certs\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.916221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.916139    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5x7\" (UniqueName: \"kubernetes.io/projected/527a8dd9-c14b-45b0-b54e-a632c59f9c1e-kube-api-access-jd5x7\") pod \"kube-auth-proxy-794f59cf45-pbxxx\" (UID: \"527a8dd9-c14b-45b0-b54e-a632c59f9c1e\") " pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:06.936021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:06.935984    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"
Oct 30 21:43:07.312341 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:07.312316    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx"]
Oct 30 21:43:07.314231 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:43:07.314195    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod527a8dd9_c14b_45b0_b54e_a632c59f9c1e.slice/crio-23a17e263025ebca6c08e06b074b0460b92b769342ddf4beb2d25cff75aece47 WatchSource:0}: Error finding container 23a17e263025ebca6c08e06b074b0460b92b769342ddf4beb2d25cff75aece47: Status 404 returned error can't find the container with id 23a17e263025ebca6c08e06b074b0460b92b769342ddf4beb2d25cff75aece47
Oct 30 21:43:07.316058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:07.316044    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:43:07.347097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:07.347054    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx" event={"ID":"527a8dd9-c14b-45b0-b54e-a632c59f9c1e","Type":"ContainerStarted","Data":"23a17e263025ebca6c08e06b074b0460b92b769342ddf4beb2d25cff75aece47"}
Oct 30 21:43:08.856705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.856606    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:08.860718 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.860687    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:08.912650 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.912621    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:08.956216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.956183    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Oct 30 21:43:08.975999 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.975958    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rplx\" (UniqueName: \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:08.976195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:08.976125    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.077799 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.076698    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.077799 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.076791    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rplx\" (UniqueName: \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.077799 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.077750    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.100624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.100570    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:09.135671 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.135596    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rplx\" (UniqueName: \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\") pod \"limitador-limitador-74dfcfb894-zzt2k\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.177553 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.177505    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:09.556853 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:09.556768    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:09.564752 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:43:09.564709    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e08283b_d344_491f_b47a_fba9bc692723.slice/crio-1f5e021c0aa211c12fd6e7544056811d2ebc11580c88c5271105f981e1be8716 WatchSource:0}: Error finding container 1f5e021c0aa211c12fd6e7544056811d2ebc11580c88c5271105f981e1be8716: Status 404 returned error can't find the container with id 1f5e021c0aa211c12fd6e7544056811d2ebc11580c88c5271105f981e1be8716
Oct 30 21:43:10.363521 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:10.363488    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" event={"ID":"1e08283b-d344-491f-b47a-fba9bc692723","Type":"ContainerStarted","Data":"1f5e021c0aa211c12fd6e7544056811d2ebc11580c88c5271105f981e1be8716"}
Oct 30 21:43:11.368842 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:11.368806    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx" event={"ID":"527a8dd9-c14b-45b0-b54e-a632c59f9c1e","Type":"ContainerStarted","Data":"3d7179ca41a8644719e95f6323795cf30502cfb3c432dc6e7498fe73ea15b49c"}
Oct 30 21:43:11.480546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:11.480485    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-794f59cf45-pbxxx" podStartSLOduration=1.858945388 podStartE2EDuration="5.480468183s" totalImagesPullingTime="3.621522795s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:11.478634528 +0000 UTC m=+314.651958956" watchObservedRunningTime="2026-10-30 21:43:11.480468183 +0000 UTC m=+314.653792607"
Oct 30 21:43:12.971375 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:12.971328    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:43:12.980577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:12.980552    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:43:12.980759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:12.980698    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:43:12.984333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:12.984300    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.027851 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.027818    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:43:13.072272 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.072235    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kd7kz\""
Oct 30 21:43:13.075378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.075347    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:43:13.116966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.116924    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-config-file\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.117152 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.117061    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfvcg\" (UniqueName: \"kubernetes.io/projected/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-kube-api-access-tfvcg\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.117152 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.117137    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5kn7\" (UniqueName: \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\") pod \"authorino-56c6c9ddd5-cp797\" (UID: \"edc33793-9b03-476e-836b-d801b9076037\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:43:13.218051 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.218014    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfvcg\" (UniqueName: \"kubernetes.io/projected/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-kube-api-access-tfvcg\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.218231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.218087    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5kn7\" (UniqueName: \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\") pod \"authorino-56c6c9ddd5-cp797\" (UID: \"edc33793-9b03-476e-836b-d801b9076037\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:43:13.218231 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.218176    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-config-file\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.218918 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.218895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-config-file\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.276869 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.276792    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfvcg\" (UniqueName: \"kubernetes.io/projected/eb806f00-3ab1-40e7-b4a2-7f7f131b7338-kube-api-access-tfvcg\") pod \"limitador-limitador-7c9b87d574-w2slh\" (UID: \"eb806f00-3ab1-40e7-b4a2-7f7f131b7338\") " pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.279134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.279107    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5kn7\" (UniqueName: \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\") pod \"authorino-56c6c9ddd5-cp797\" (UID: \"edc33793-9b03-476e-836b-d801b9076037\") " pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:43:13.295630 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.295573    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:43:13.303362 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.303037    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:13.304122 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.304091    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:43:13.674756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.674735    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:43:13.677441 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:43:13.677415    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podedc33793_9b03_476e_836b_d801b9076037.slice/crio-1ea328c1cdbe19d860ee745665a938b26306981ee4c711021a415a8e9263593f WatchSource:0}: Error finding container 1ea328c1cdbe19d860ee745665a938b26306981ee4c711021a415a8e9263593f: Status 404 returned error can't find the container with id 1ea328c1cdbe19d860ee745665a938b26306981ee4c711021a415a8e9263593f
Oct 30 21:43:13.723482 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:13.723461    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:43:13.725698 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:43:13.725665    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb806f00_3ab1_40e7_b4a2_7f7f131b7338.slice/crio-d348079397e6a639e6f981e36f9c8308b585e8c286c784d7bb178648f810ce6a WatchSource:0}: Error finding container d348079397e6a639e6f981e36f9c8308b585e8c286c784d7bb178648f810ce6a: Status 404 returned error can't find the container with id d348079397e6a639e6f981e36f9c8308b585e8c286c784d7bb178648f810ce6a
Oct 30 21:43:14.387370 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.387337    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" event={"ID":"1e08283b-d344-491f-b47a-fba9bc692723","Type":"ContainerStarted","Data":"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"}
Oct 30 21:43:14.387846 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.387524    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:14.388678 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.388648    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" event={"ID":"edc33793-9b03-476e-836b-d801b9076037","Type":"ContainerStarted","Data":"1ea328c1cdbe19d860ee745665a938b26306981ee4c711021a415a8e9263593f"}
Oct 30 21:43:14.390180 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.390159    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh" event={"ID":"eb806f00-3ab1-40e7-b4a2-7f7f131b7338","Type":"ContainerStarted","Data":"8f464983fda1f5bc4b7592068242745787422efbc167a4cf8ec1a43bbc4b14c7"}
Oct 30 21:43:14.390256 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.390198    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh" event={"ID":"eb806f00-3ab1-40e7-b4a2-7f7f131b7338","Type":"ContainerStarted","Data":"d348079397e6a639e6f981e36f9c8308b585e8c286c784d7bb178648f810ce6a"}
Oct 30 21:43:14.390306 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.390287    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:14.498230 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.498174    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" podStartSLOduration=2.456849492 podStartE2EDuration="6.498158028s" totalImagesPullingTime="4.041308536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:14.496713745 +0000 UTC m=+317.670038172" watchObservedRunningTime="2026-10-30 21:43:14.498158028 +0000 UTC m=+317.671482455"
Oct 30 21:43:14.614397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:14.614336    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh" podStartSLOduration=2.614320639 podStartE2EDuration="2.614320639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:14.613377077 +0000 UTC m=+317.786701502" watchObservedRunningTime="2026-10-30 21:43:14.614320639 +0000 UTC m=+317.787645065"
Oct 30 21:43:16.404647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:16.404610    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" event={"ID":"edc33793-9b03-476e-836b-d801b9076037","Type":"ContainerStarted","Data":"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"}
Oct 30 21:43:16.512731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:16.512679    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" podStartSLOduration=1.880245747 podStartE2EDuration="4.512664773s" totalImagesPullingTime="2.632419026s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:16.511855555 +0000 UTC m=+319.685179978" watchObservedRunningTime="2026-10-30 21:43:16.512664773 +0000 UTC m=+319.685989197"
Oct 30 21:43:20.806191 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:43:20.806151    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:43:25.399267 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:25.399234    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:25.399918 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:25.399316    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-7c9b87d574-w2slh"
Oct 30 21:43:25.650117 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:25.650025    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:25.650288 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:25.650257    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" podUID="1e08283b-d344-491f-b47a-fba9bc692723" containerName="limitador" containerID="cri-o://42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec" gracePeriod=30
Oct 30 21:43:26.191038 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.191013    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:26.232036 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.231952    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\" (UniqueName: \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\") pod \"1e08283b-d344-491f-b47a-fba9bc692723\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") "
Oct 30 21:43:26.232036 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.231998    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\" (UniqueName: \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\") pod \"1e08283b-d344-491f-b47a-fba9bc692723\" (UID: \"1e08283b-d344-491f-b47a-fba9bc692723\") "
Oct 30 21:43:26.232320 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.232289    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file" pod "1e08283b-d344-491f-b47a-fba9bc692723" (UID: "1e08283b-d344-491f-b47a-fba9bc692723"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:43:26.234145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.234122    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx" pod "1e08283b-d344-491f-b47a-fba9bc692723" (UID: "1e08283b-d344-491f-b47a-fba9bc692723"). InnerVolumeSpecName "kube-api-access-5rplx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:43:26.332982 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.332945    4882 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/1e08283b-d344-491f-b47a-fba9bc692723-config-file\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:26.332982 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.332976    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5rplx\" (UniqueName: \"kubernetes.io/projected/1e08283b-d344-491f-b47a-fba9bc692723-kube-api-access-5rplx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:43:26.483355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.483273    4882 generic.go:356] "Generic (PLEG): container finished" podID="1e08283b-d344-491f-b47a-fba9bc692723" containerID="42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec" exitCode=0
Oct 30 21:43:26.483355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.483324    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"
Oct 30 21:43:26.483845 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.483355    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" event={"ID":"1e08283b-d344-491f-b47a-fba9bc692723","Type":"ContainerDied","Data":"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"}
Oct 30 21:43:26.483845 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.483414    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-zzt2k" event={"ID":"1e08283b-d344-491f-b47a-fba9bc692723","Type":"ContainerDied","Data":"1f5e021c0aa211c12fd6e7544056811d2ebc11580c88c5271105f981e1be8716"}
Oct 30 21:43:26.483845 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.483436    4882 scope.go:122] "RemoveContainer" containerID="42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"
Oct 30 21:43:26.492466 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.492447    4882 scope.go:122] "RemoveContainer" containerID="42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"
Oct 30 21:43:26.492787 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:43:26.492768    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec\": container with ID starting with 42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec not found: ID does not exist" containerID="42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"
Oct 30 21:43:26.492854 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.492796    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec"} err="failed to get container status \"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec\": rpc error: code = NotFound desc = could not find container \"42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec\": container with ID starting with 42947599b118ee0f21a249314bb49ffa444bf43dbc6fbd066163768f19b816ec not found: ID does not exist"
Oct 30 21:43:26.637351 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.637320    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:26.644108 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.644078    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-zzt2k"]
Oct 30 21:43:26.941526 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:26.941492    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e08283b-d344-491f-b47a-fba9bc692723" path="/var/lib/kubelet/pods/1e08283b-d344-491f-b47a-fba9bc692723/volumes"
Oct 30 21:43:32.837612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.837553    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"]
Oct 30 21:43:32.838157 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.838036    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1e08283b-d344-491f-b47a-fba9bc692723" containerName="limitador"
Oct 30 21:43:32.838157 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.838048    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e08283b-d344-491f-b47a-fba9bc692723" containerName="limitador"
Oct 30 21:43:32.838157 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.838143    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e08283b-d344-491f-b47a-fba9bc692723" containerName="limitador"
Oct 30 21:43:32.841373 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.841352    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:32.892842 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.892800    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2qll\" (UniqueName: \"kubernetes.io/projected/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-kube-api-access-p2qll\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:32.893012 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.892867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-data\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:32.900625 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.900577    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"]
Oct 30 21:43:32.945032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.944995    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Oct 30 21:43:32.945032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.945028    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Oct 30 21:43:32.945330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.945051    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Oct 30 21:43:32.945330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.945060    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-k8qm2\""
Oct 30 21:43:32.993432 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.993390    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2qll\" (UniqueName: \"kubernetes.io/projected/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-kube-api-access-p2qll\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:32.993687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.993458    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-data\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:32.993956 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:32.993935    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-data\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:33.051439 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:33.051409    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2qll\" (UniqueName: \"kubernetes.io/projected/3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2-kube-api-access-p2qll\") pod \"postgres-7cdbf6dc44-kzzxd\" (UID: \"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:33.154659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:33.154544    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:33.527013 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:33.526947    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"]
Oct 30 21:43:33.529425 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:43:33.529398    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e7d26f3_57e7_4aa4_9ac2_75ac97d1ecf2.slice/crio-7ec8f883797c37b6a35e8893f147aa38a9bd93eb54d8fb8a71295bf01f7e4c14 WatchSource:0}: Error finding container 7ec8f883797c37b6a35e8893f147aa38a9bd93eb54d8fb8a71295bf01f7e4c14: Status 404 returned error can't find the container with id 7ec8f883797c37b6a35e8893f147aa38a9bd93eb54d8fb8a71295bf01f7e4c14
Oct 30 21:43:34.523155 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:34.523117    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd" event={"ID":"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2","Type":"ContainerStarted","Data":"7ec8f883797c37b6a35e8893f147aa38a9bd93eb54d8fb8a71295bf01f7e4c14"}
Oct 30 21:43:40.555294 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:40.555240    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd" event={"ID":"3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2","Type":"ContainerStarted","Data":"0b7bdee998b83d7bdc5aa76de3be8862de762856956bfd0586e29754b4dd7834"}
Oct 30 21:43:40.555714 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:40.555312    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:40.665260 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:40.665207    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd" podStartSLOduration=2.135687235 podStartE2EDuration="8.66519379s" totalImagesPullingTime="6.529506555s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:43:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:43:40.664180172 +0000 UTC m=+343.837504598" watchObservedRunningTime="2026-10-30 21:43:40.66519379 +0000 UTC m=+343.838518238"
Oct 30 21:43:46.589674 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:46.589644    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-kzzxd"
Oct 30 21:43:50.805613 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:43:50.805557    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:43:56.964750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:43:56.964716    4882 scope.go:122] "RemoveContainer" containerID="9ce770983b879bc3f104f1e576b97c68ab4337aa2a8da56dc42cdba4bcc3798c"
Oct 30 21:44:20.807271 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:44:20.807226    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:44:24.804661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.804617    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bzpq4"]
Oct 30 21:44:24.828268 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.828232    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:24.868175 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.868148    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bzpq4"]
Oct 30 21:44:24.881333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.881301    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-cert\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:24.881505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.881367    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24cxx\" (UniqueName: \"kubernetes.io/projected/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-kube-api-access-24cxx\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:24.912778 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.912741    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Oct 30 21:44:24.924798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.924758    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Oct 30 21:44:24.924798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.924774    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-skzmz\""
Oct 30 21:44:24.924798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.924794    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Oct 30 21:44:24.926750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.926723    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:44:24.930383 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.930362    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:24.977109 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.977079    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:44:24.979819 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.979798    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Oct 30 21:44:24.979819 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.979799    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-lwxbm\""
Oct 30 21:44:24.982280 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.982252    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-cert\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:24.982416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.982296    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct46t\" (UniqueName: \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:24.982416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.982338    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24cxx\" (UniqueName: \"kubernetes.io/projected/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-kube-api-access-24cxx\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:24.982416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.982355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:24.984668 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:24.984647    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-cert\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:25.020736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.020706    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bmxth"]
Oct 30 21:44:25.024413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.024394    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.044535 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.044508    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24cxx\" (UniqueName: \"kubernetes.io/projected/3fc14e3f-e24b-4525-b02f-f762c0f2afc8-kube-api-access-24cxx\") pod \"kserve-controller-manager-776cc66d47-bzpq4\" (UID: \"3fc14e3f-e24b-4525-b02f-f762c0f2afc8\") " pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:25.080236 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.080153    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bmxth"]
Oct 30 21:44:25.080385 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.080249    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m9z5l\""
Oct 30 21:44:25.080385 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.080273    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Oct 30 21:44:25.083549 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.083524    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct46t\" (UniqueName: \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:25.083729 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.083621    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:25.083729 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.083699    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3c8bce-2e78-4a51-8636-8058e2857a96-tls-certs\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.083849 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.083730    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7hv7\" (UniqueName: \"kubernetes.io/projected/9f3c8bce-2e78-4a51-8636-8058e2857a96-kube-api-access-f7hv7\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.086166 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.086145    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:25.136759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.136726    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-qx8nl"]
Oct 30 21:44:25.140531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.140510    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.141483 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.141463    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:25.145225 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.145202    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct46t\" (UniqueName: \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\") pod \"llmisvc-controller-manager-869d5dcc5d-xtzhq\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") " pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:25.184361 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.184326    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-metrics-certs\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.184544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.184369    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-cert\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.184544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.184431    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vkfnp\" (UniqueName: \"kubernetes.io/projected/a946fc79-675f-41f6-b52f-882569f35dc2-kube-api-access-vkfnp\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.184544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.184467    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3c8bce-2e78-4a51-8636-8058e2857a96-tls-certs\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.184544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.184486    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7hv7\" (UniqueName: \"kubernetes.io/projected/9f3c8bce-2e78-4a51-8636-8058e2857a96-kube-api-access-f7hv7\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.186972 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.186939    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f3c8bce-2e78-4a51-8636-8058e2857a96-tls-certs\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.194047 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.192883    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-btqrl\""
Oct 30 21:44:25.194047 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.193513    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Oct 30 21:44:25.194047 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.193570    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\""
Oct 30 21:44:25.196902 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.196701    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qx8nl"]
Oct 30 21:44:25.242552 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.242520    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7hv7\" (UniqueName: \"kubernetes.io/projected/9f3c8bce-2e78-4a51-8636-8058e2857a96-kube-api-access-f7hv7\") pod \"model-serving-api-75bd89fdfc-bmxth\" (UID: \"9f3c8bce-2e78-4a51-8636-8058e2857a96\") " pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.242714 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.242687    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:25.285190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.285155    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-metrics-certs\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.285363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.285203    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-cert\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.285363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.285226    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vkfnp\" (UniqueName: \"kubernetes.io/projected/a946fc79-675f-41f6-b52f-882569f35dc2-kube-api-access-vkfnp\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.287735 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.287708    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-metrics-certs\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.287877 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.287784    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a946fc79-675f-41f6-b52f-882569f35dc2-cert\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.338194 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.338111    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:25.343432 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.343407    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vkfnp\" (UniqueName: \"kubernetes.io/projected/a946fc79-675f-41f6-b52f-882569f35dc2-kube-api-access-vkfnp\") pod \"odh-model-controller-769875db56-qx8nl\" (UID: \"a946fc79-675f-41f6-b52f-882569f35dc2\") " pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.477114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.477078    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:25.511462 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.511430    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-bzpq4"]
Oct 30 21:44:25.512925 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:44:25.512896    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fc14e3f_e24b_4525_b02f_f762c0f2afc8.slice/crio-f4e627fc4c9e6bb550fd469f4fbaf119406dd364487bea9d5ac5549b3f0ee8af WatchSource:0}: Error finding container f4e627fc4c9e6bb550fd469f4fbaf119406dd364487bea9d5ac5549b3f0ee8af: Status 404 returned error can't find the container with id f4e627fc4c9e6bb550fd469f4fbaf119406dd364487bea9d5ac5549b3f0ee8af
Oct 30 21:44:25.607693 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.607667    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:44:25.608856 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:44:25.608828    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod72349f10_60de_4b70_a123_ef9f8f9d42ec.slice/crio-4b88bf928d0673ea95f78ddcc5cc681478ea9fb5ae12b897e6025f7bf686f369 WatchSource:0}: Error finding container 4b88bf928d0673ea95f78ddcc5cc681478ea9fb5ae12b897e6025f7bf686f369: Status 404 returned error can't find the container with id 4b88bf928d0673ea95f78ddcc5cc681478ea9fb5ae12b897e6025f7bf686f369
Oct 30 21:44:25.703444 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.703419    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-bmxth"]
Oct 30 21:44:25.705306 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:44:25.705277    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f3c8bce_2e78_4a51_8636_8058e2857a96.slice/crio-87566ef94ef1871a58aeec994e45306dfe9faaee48e3ecce39d74d82d064f11e WatchSource:0}: Error finding container 87566ef94ef1871a58aeec994e45306dfe9faaee48e3ecce39d74d82d064f11e: Status 404 returned error can't find the container with id 87566ef94ef1871a58aeec994e45306dfe9faaee48e3ecce39d74d82d064f11e
Oct 30 21:44:25.760764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.760719    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4" event={"ID":"3fc14e3f-e24b-4525-b02f-f762c0f2afc8","Type":"ContainerStarted","Data":"f4e627fc4c9e6bb550fd469f4fbaf119406dd364487bea9d5ac5549b3f0ee8af"}
Oct 30 21:44:25.761943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.761911    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" event={"ID":"72349f10-60de-4b70-a123-ef9f8f9d42ec","Type":"ContainerStarted","Data":"4b88bf928d0673ea95f78ddcc5cc681478ea9fb5ae12b897e6025f7bf686f369"}
Oct 30 21:44:25.763034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.763008    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth" event={"ID":"9f3c8bce-2e78-4a51-8636-8058e2857a96","Type":"ContainerStarted","Data":"87566ef94ef1871a58aeec994e45306dfe9faaee48e3ecce39d74d82d064f11e"}
Oct 30 21:44:25.850329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:25.850301    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-qx8nl"]
Oct 30 21:44:25.851834 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:44:25.851806    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda946fc79_675f_41f6_b52f_882569f35dc2.slice/crio-ae81b10d0d0b9df4ad0572db0f41a039935dc5479d208c5c9dfeda6465e77685 WatchSource:0}: Error finding container ae81b10d0d0b9df4ad0572db0f41a039935dc5479d208c5c9dfeda6465e77685: Status 404 returned error can't find the container with id ae81b10d0d0b9df4ad0572db0f41a039935dc5479d208c5c9dfeda6465e77685
Oct 30 21:44:26.770832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:26.770784    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qx8nl" event={"ID":"a946fc79-675f-41f6-b52f-882569f35dc2","Type":"ContainerStarted","Data":"ae81b10d0d0b9df4ad0572db0f41a039935dc5479d208c5c9dfeda6465e77685"}
Oct 30 21:44:28.015392 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.015352    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"]
Oct 30 21:44:28.019986 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.019852    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.071378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.071342    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\""
Oct 30 21:44:28.113937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.113873    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.114118 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.113949    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2x87\" (UniqueName: \"kubernetes.io/projected/c39beeb8-af5b-4396-be2b-80d1c65fb869-kube-api-access-j2x87\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.114118 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.114053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c39beeb8-af5b-4396-be2b-80d1c65fb869-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.114118 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.114089    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.119945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.118735    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\""
Oct 30 21:44:28.119945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.118815    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-2j76j\""
Oct 30 21:44:28.119945 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.118864    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\""
Oct 30 21:44:28.122463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.122436    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"]
Oct 30 21:44:28.214718 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.214657    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.214718 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.214703    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2x87\" (UniqueName: \"kubernetes.io/projected/c39beeb8-af5b-4396-be2b-80d1c65fb869-kube-api-access-j2x87\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.215044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.214786    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c39beeb8-af5b-4396-be2b-80d1c65fb869-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.215044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.214912    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.215044 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:44:28.215034    4882 configmap.go:193] Couldn't get configMap opendatahub/usage-tenancy-proxy-source: configmap "usage-tenancy-proxy-source" not found
Oct 30 21:44:28.215209 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:44:28.215121    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source podName:c39beeb8-af5b-4396-be2b-80d1c65fb869 nodeName:}" failed. No retries permitted until 2026-10-30 21:44:28.715097664 +0000 UTC m=+391.888422078 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-source" (UniqueName: "kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source") pod "usage-logs-tenancy-proxy-9c5d49bcd-929tg" (UID: "c39beeb8-af5b-4396-be2b-80d1c65fb869") : configmap "usage-tenancy-proxy-source" not found
Oct 30 21:44:28.215446 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.215426    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.222238 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.222204    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/c39beeb8-af5b-4396-be2b-80d1c65fb869-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.275337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.275267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2x87\" (UniqueName: \"kubernetes.io/projected/c39beeb8-af5b-4396-be2b-80d1c65fb869-kube-api-access-j2x87\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.721229 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.721181    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.722447 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.722358    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/c39beeb8-af5b-4396-be2b-80d1c65fb869-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-929tg\" (UID: \"c39beeb8-af5b-4396-be2b-80d1c65fb869\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:28.939545 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:28.939292    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:29.332318 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.332288    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"]
Oct 30 21:44:29.792669 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.792627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4" event={"ID":"3fc14e3f-e24b-4525-b02f-f762c0f2afc8","Type":"ContainerStarted","Data":"bdb2660db0e83b87f758d032c98e3f1cdf1f074547891b45d3d3a5ed26d29331"}
Oct 30 21:44:29.792854 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.792781    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:44:29.794531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.794496    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" event={"ID":"72349f10-60de-4b70-a123-ef9f8f9d42ec","Type":"ContainerStarted","Data":"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"}
Oct 30 21:44:29.794688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.794561    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:44:29.796291 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.796262    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth" event={"ID":"9f3c8bce-2e78-4a51-8636-8058e2857a96","Type":"ContainerStarted","Data":"105a8a1225ad86f8a049bb95641067562933c30a436d1ff08079b07be0145f9c"}
Oct 30 21:44:29.796414 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.796314    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:29.797565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:29.797543    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg" event={"ID":"c39beeb8-af5b-4396-be2b-80d1c65fb869","Type":"ContainerStarted","Data":"be9c9dd4af5c2d55d8e4fbc5f968a1e6f9039324afaa957970171174792ec88d"}
Oct 30 21:44:30.040741 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:30.040688    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" podStartSLOduration=2.347082898 podStartE2EDuration="6.040673071s" totalImagesPullingTime="3.693590173s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.040099398 +0000 UTC m=+393.213423823" watchObservedRunningTime="2026-10-30 21:44:30.040673071 +0000 UTC m=+393.213997495"
Oct 30 21:44:30.041120 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:30.041085    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4" podStartSLOduration=2.408236022 podStartE2EDuration="6.041075244s" totalImagesPullingTime="3.632839222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:29.9243321 +0000 UTC m=+393.097656525" watchObservedRunningTime="2026-10-30 21:44:30.041075244 +0000 UTC m=+393.214399670"
Oct 30 21:44:30.153638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:30.153572    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth" podStartSLOduration=2.83594325 podStartE2EDuration="6.153556325s" totalImagesPullingTime="3.317613075s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.151530589 +0000 UTC m=+393.324855015" watchObservedRunningTime="2026-10-30 21:44:30.153556325 +0000 UTC m=+393.326880810"
Oct 30 21:44:30.803361 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:30.803317    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-qx8nl" event={"ID":"a946fc79-675f-41f6-b52f-882569f35dc2","Type":"ContainerStarted","Data":"d40c78d398b7b705adc95e1b2b8d4d52e747d95529e6de214b336c802e4ef665"}
Oct 30 21:44:30.915029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:30.914974    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-qx8nl" podStartSLOduration=1.7621902409999999 podStartE2EDuration="5.914958732s" totalImagesPullingTime="4.152768491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:30.914350151 +0000 UTC m=+394.087674578" watchObservedRunningTime="2026-10-30 21:44:30.914958732 +0000 UTC m=+394.088283199"
Oct 30 21:44:31.816995 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:31.816967    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:40.808210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:40.808179    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-bmxth"
Oct 30 21:44:42.825758 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:42.825727    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-qx8nl"
Oct 30 21:44:42.879173 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:42.879137    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg" event={"ID":"c39beeb8-af5b-4396-be2b-80d1c65fb869","Type":"ContainerStarted","Data":"f8f6f4126680ee3f882962dc9bcce20a33e87fc25353ff5bf28e772875d722c7"}
Oct 30 21:44:42.879385 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:42.879208    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:44:43.055735 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:43.055678    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg" podStartSLOduration=3.1431701690000002 podStartE2EDuration="16.055664424s" totalImagesPullingTime="12.912494255s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:44:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:44:43.054020019 +0000 UTC m=+406.227344446" watchObservedRunningTime="2026-10-30 21:44:43.055664424 +0000 UTC m=+406.228988850"
Oct 30 21:44:50.805637 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:44:50.805579    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:44:53.889703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:44:53.889669    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-929tg"
Oct 30 21:45:00.170351 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.170305    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:00.257702 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.257663    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:00.257894 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.257769    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.324182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.324149    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ft4t\" (UniqueName: \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.324182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.324186    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.363895 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.363862    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:45:00.425405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.425320    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ft4t\" (UniqueName: \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.425405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.425362    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.426090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.426070    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.487249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.487212    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ft4t\" (UniqueName: \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\") pod \"maas-api-key-cleanup-29889945-w87lt\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.578582 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.578547    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:00.806263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.806171    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:45:00.809334 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:00.809307    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-bzpq4"
Oct 30 21:45:01.029449 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:45:01.029407    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20538d23_6213_4385_ab16_74170d47532e.slice/crio-af29cde77a8d8f178e2dbd85584d6b02cd85319bc9c7a9d38412b08f9eb4ec6f WatchSource:0}: Error finding container af29cde77a8d8f178e2dbd85584d6b02cd85319bc9c7a9d38412b08f9eb4ec6f: Status 404 returned error can't find the container with id af29cde77a8d8f178e2dbd85584d6b02cd85319bc9c7a9d38412b08f9eb4ec6f
Oct 30 21:45:01.031156 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:01.031134    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:01.976290 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:01.976252    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" event={"ID":"20538d23-6213-4385-ab16-74170d47532e","Type":"ContainerStarted","Data":"af29cde77a8d8f178e2dbd85584d6b02cd85319bc9c7a9d38412b08f9eb4ec6f"}
Oct 30 21:45:05.141463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:05.141425    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:45:07.002109 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:07.002074    4882 generic.go:356] "Generic (PLEG): container finished" podID="20538d23-6213-4385-ab16-74170d47532e" containerID="7eef5bc226fac759416f6b4f9e3faef4a64dff6942a25a464eb2d0586b044fff" exitCode=7
Oct 30 21:45:07.002109 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:07.002115    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" event={"ID":"20538d23-6213-4385-ab16-74170d47532e","Type":"ContainerDied","Data":"7eef5bc226fac759416f6b4f9e3faef4a64dff6942a25a464eb2d0586b044fff"}
Oct 30 21:45:07.002788 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:07.002438    4882 scope.go:122] "RemoveContainer" containerID="7eef5bc226fac759416f6b4f9e3faef4a64dff6942a25a464eb2d0586b044fff"
Oct 30 21:45:08.008806 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:08.008732    4882 generic.go:356] "Generic (PLEG): container finished" podID="20538d23-6213-4385-ab16-74170d47532e" containerID="28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b" exitCode=7
Oct 30 21:45:08.009191 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:08.008860    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" event={"ID":"20538d23-6213-4385-ab16-74170d47532e","Type":"ContainerDied","Data":"28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b"}
Oct 30 21:45:08.009191 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:08.008902    4882 scope.go:122] "RemoveContainer" containerID="7eef5bc226fac759416f6b4f9e3faef4a64dff6942a25a464eb2d0586b044fff"
Oct 30 21:45:08.009191 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:08.009123    4882 scope.go:122] "RemoveContainer" containerID="28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b"
Oct 30 21:45:08.009380 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:08.009356    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-w87lt_odh-ai-gateway-infra(20538d23-6213-4385-ab16-74170d47532e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" podUID="20538d23-6213-4385-ab16-74170d47532e"
Oct 30 21:45:09.016167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:09.016141    4882 scope.go:122] "RemoveContainer" containerID="28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b"
Oct 30 21:45:09.016574 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:09.016354    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-w87lt_odh-ai-gateway-infra(20538d23-6213-4385-ab16-74170d47532e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" podUID="20538d23-6213-4385-ab16-74170d47532e"
Oct 30 21:45:13.889858 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:13.889814    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:13.892670 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:13.892649    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:13.945700 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:13.945663    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:13.952804 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:13.952776    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9xtj\" (UniqueName: \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\") pod \"authorino-6f6944bcdd-ljvv5\" (UID: \"a40a0c2c-6943-4572-a7f3-5a4388f54afe\") " pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:14.053970 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.053937    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9xtj\" (UniqueName: \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\") pod \"authorino-6f6944bcdd-ljvv5\" (UID: \"a40a0c2c-6943-4572-a7f3-5a4388f54afe\") " pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:14.118990 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.118959    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9xtj\" (UniqueName: \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\") pod \"authorino-6f6944bcdd-ljvv5\" (UID: \"a40a0c2c-6943-4572-a7f3-5a4388f54afe\") " pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:14.203129 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.203030    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:14.260172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.260143    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:14.291328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.291295    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:14.294488 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.294471    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:14.349759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.349726    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:14.356689 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.356657    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w7mh2\" (UniqueName: \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\") pod \"authorino-6754bbb7d6-nf8jh\" (UID: \"ef6bcba2-a205-4a03-8b97-6d60105ab6aa\") " pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:14.457505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.457419    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w7mh2\" (UniqueName: \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\") pod \"authorino-6754bbb7d6-nf8jh\" (UID: \"ef6bcba2-a205-4a03-8b97-6d60105ab6aa\") " pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:14.519256 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.519220    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w7mh2\" (UniqueName: \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\") pod \"authorino-6754bbb7d6-nf8jh\" (UID: \"ef6bcba2-a205-4a03-8b97-6d60105ab6aa\") " pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:14.569300 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:45:14.569256    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda40a0c2c_6943_4572_a7f3_5a4388f54afe.slice/crio-d41a6c8cb9e58cb566350aa5e84d827b49f5a9ba64c802ce578aa1c90924a7a5 WatchSource:0}: Error finding container d41a6c8cb9e58cb566350aa5e84d827b49f5a9ba64c802ce578aa1c90924a7a5: Status 404 returned error can't find the container with id d41a6c8cb9e58cb566350aa5e84d827b49f5a9ba64c802ce578aa1c90924a7a5
Oct 30 21:45:14.570614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.570574    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:14.606334 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.606297    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:14.720406 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.720328    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:14.751440 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.751410    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:45:14.754704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.754685    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.808158 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.808132    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Oct 30 21:45:14.809479 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.809456    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:45:14.860445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.860414    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.860625 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.860565    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zvp2k\" (UniqueName: \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.961826 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.961792    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zvp2k\" (UniqueName: \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.962267 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.961846    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.964284 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.964258    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:14.972184 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:45:14.972114    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podef6bcba2_a205_4a03_8b97_6d60105ab6aa.slice/crio-ce9c47bf66a84a7c48beb8a9240ccaffdab191260b60eb15c7fea0d53f3fe7f6 WatchSource:0}: Error finding container ce9c47bf66a84a7c48beb8a9240ccaffdab191260b60eb15c7fea0d53f3fe7f6: Status 404 returned error can't find the container with id ce9c47bf66a84a7c48beb8a9240ccaffdab191260b60eb15c7fea0d53f3fe7f6
Oct 30 21:45:14.974351 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:14.974325    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:15.025125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:15.025095    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zvp2k\" (UniqueName: \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\") pod \"authorino-6d6676b56-w5w5l\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") " pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:15.045526 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:15.045484    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" event={"ID":"a40a0c2c-6943-4572-a7f3-5a4388f54afe","Type":"ContainerStarted","Data":"d41a6c8cb9e58cb566350aa5e84d827b49f5a9ba64c802ce578aa1c90924a7a5"}
Oct 30 21:45:15.046825 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:15.046802    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" event={"ID":"ef6bcba2-a205-4a03-8b97-6d60105ab6aa","Type":"ContainerStarted","Data":"ce9c47bf66a84a7c48beb8a9240ccaffdab191260b60eb15c7fea0d53f3fe7f6"}
Oct 30 21:45:15.066203 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:15.066179    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:45:15.456185 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:45:15.456151    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod349015b9_cd29_44f3_b5fc_7360f1f5437b.slice/crio-c5f7373e22175a0de2fddfbaf939386a0c9f20c165b6a49fbb0a8eab5b72607b WatchSource:0}: Error finding container c5f7373e22175a0de2fddfbaf939386a0c9f20c165b6a49fbb0a8eab5b72607b: Status 404 returned error can't find the container with id c5f7373e22175a0de2fddfbaf939386a0c9f20c165b6a49fbb0a8eab5b72607b
Oct 30 21:45:15.457655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:15.457631    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:45:16.053066 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:16.053033    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6676b56-w5w5l" event={"ID":"349015b9-cd29-44f3-b5fc-7360f1f5437b","Type":"ContainerStarted","Data":"c5f7373e22175a0de2fddfbaf939386a0c9f20c165b6a49fbb0a8eab5b72607b"}
Oct 30 21:45:17.064628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.063676    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" event={"ID":"ef6bcba2-a205-4a03-8b97-6d60105ab6aa","Type":"ContainerStarted","Data":"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"}
Oct 30 21:45:17.064628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.063757    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" podUID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" containerName="authorino" containerID="cri-o://62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28" gracePeriod=30
Oct 30 21:45:17.066007 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.065974    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6676b56-w5w5l" event={"ID":"349015b9-cd29-44f3-b5fc-7360f1f5437b","Type":"ContainerStarted","Data":"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"}
Oct 30 21:45:17.068927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.068705    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" podUID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" containerName="authorino" containerID="cri-o://168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b" gracePeriod=30
Oct 30 21:45:17.068927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.068717    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" event={"ID":"a40a0c2c-6943-4572-a7f3-5a4388f54afe","Type":"ContainerStarted","Data":"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"}
Oct 30 21:45:17.182573 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.182508    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" podStartSLOduration=1.701311065 podStartE2EDuration="3.182487343s" totalImagesPullingTime="1.481176278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:17.179731409 +0000 UTC m=+440.353055835" watchObservedRunningTime="2026-10-30 21:45:17.182487343 +0000 UTC m=+440.355811771"
Oct 30 21:45:17.298249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.298161    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" podStartSLOduration=2.705212945 podStartE2EDuration="4.298141966s" totalImagesPullingTime="1.592929021s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:17.29771732 +0000 UTC m=+440.471041746" watchObservedRunningTime="2026-10-30 21:45:17.298141966 +0000 UTC m=+440.471466395"
Oct 30 21:45:17.356395 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.356364    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:17.359050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.359019    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:17.417318 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.417253    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6d6676b56-w5w5l" podStartSLOduration=1.91810693 podStartE2EDuration="3.41723533s" totalImagesPullingTime="1.4991284s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:17.416470795 +0000 UTC m=+440.589795221" watchObservedRunningTime="2026-10-30 21:45:17.41723533 +0000 UTC m=+440.590559758"
Oct 30 21:45:17.452164 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.452116    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:45:17.452622 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.452577    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" podUID="edc33793-9b03-476e-836b-d801b9076037" containerName="authorino" containerID="cri-o://4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67" gracePeriod=30
Oct 30 21:45:17.486894 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.486870    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\" (UniqueName: \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\") pod \"a40a0c2c-6943-4572-a7f3-5a4388f54afe\" (UID: \"a40a0c2c-6943-4572-a7f3-5a4388f54afe\") "
Oct 30 21:45:17.487054 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.486932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\" (UniqueName: \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\") pod \"ef6bcba2-a205-4a03-8b97-6d60105ab6aa\" (UID: \"ef6bcba2-a205-4a03-8b97-6d60105ab6aa\") "
Oct 30 21:45:17.489362 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.489266    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2" pod "ef6bcba2-a205-4a03-8b97-6d60105ab6aa" (UID: "ef6bcba2-a205-4a03-8b97-6d60105ab6aa"). InnerVolumeSpecName "kube-api-access-w7mh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:17.489487 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.489447    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj" pod "a40a0c2c-6943-4572-a7f3-5a4388f54afe" (UID: "a40a0c2c-6943-4572-a7f3-5a4388f54afe"). InnerVolumeSpecName "kube-api-access-d9xtj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:17.588227 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.588190    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w7mh2\" (UniqueName: \"kubernetes.io/projected/ef6bcba2-a205-4a03-8b97-6d60105ab6aa-kube-api-access-w7mh2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:17.588227 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.588225    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9xtj\" (UniqueName: \"kubernetes.io/projected/a40a0c2c-6943-4572-a7f3-5a4388f54afe-kube-api-access-d9xtj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:17.696474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.696445    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:45:17.790184 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.790147    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\" (UniqueName: \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\") pod \"edc33793-9b03-476e-836b-d801b9076037\" (UID: \"edc33793-9b03-476e-836b-d801b9076037\") "
Oct 30 21:45:17.792222 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.792197    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7" pod "edc33793-9b03-476e-836b-d801b9076037" (UID: "edc33793-9b03-476e-836b-d801b9076037"). InnerVolumeSpecName "kube-api-access-c5kn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:17.891261 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:17.891227    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5kn7\" (UniqueName: \"kubernetes.io/projected/edc33793-9b03-476e-836b-d801b9076037-kube-api-access-c5kn7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:18.075516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.075480    4882 generic.go:356] "Generic (PLEG): container finished" podID="edc33793-9b03-476e-836b-d801b9076037" containerID="4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67" exitCode=0
Oct 30 21:45:18.075979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.075547    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-cp797"
Oct 30 21:45:18.075979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.075573    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" event={"ID":"edc33793-9b03-476e-836b-d801b9076037","Type":"ContainerDied","Data":"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"}
Oct 30 21:45:18.075979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.075629    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-cp797" event={"ID":"edc33793-9b03-476e-836b-d801b9076037","Type":"ContainerDied","Data":"1ea328c1cdbe19d860ee745665a938b26306981ee4c711021a415a8e9263593f"}
Oct 30 21:45:18.075979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.075652    4882 scope.go:122] "RemoveContainer" containerID="4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"
Oct 30 21:45:18.077016 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.076988    4882 generic.go:356] "Generic (PLEG): container finished" podID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" containerID="62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28" exitCode=0
Oct 30 21:45:18.077097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.077028    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" event={"ID":"ef6bcba2-a205-4a03-8b97-6d60105ab6aa","Type":"ContainerDied","Data":"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"}
Oct 30 21:45:18.077097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.077037    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh"
Oct 30 21:45:18.077097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.077060    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6754bbb7d6-nf8jh" event={"ID":"ef6bcba2-a205-4a03-8b97-6d60105ab6aa","Type":"ContainerDied","Data":"ce9c47bf66a84a7c48beb8a9240ccaffdab191260b60eb15c7fea0d53f3fe7f6"}
Oct 30 21:45:18.078643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.078599    4882 generic.go:356] "Generic (PLEG): container finished" podID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" containerID="168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b" exitCode=0
Oct 30 21:45:18.078760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.078663    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5"
Oct 30 21:45:18.078760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.078703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" event={"ID":"a40a0c2c-6943-4572-a7f3-5a4388f54afe","Type":"ContainerDied","Data":"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"}
Oct 30 21:45:18.078760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.078744    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f6944bcdd-ljvv5" event={"ID":"a40a0c2c-6943-4572-a7f3-5a4388f54afe","Type":"ContainerDied","Data":"d41a6c8cb9e58cb566350aa5e84d827b49f5a9ba64c802ce578aa1c90924a7a5"}
Oct 30 21:45:18.086997 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.086975    4882 scope.go:122] "RemoveContainer" containerID="4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"
Oct 30 21:45:18.087335 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:18.087314    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67\": container with ID starting with 4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67 not found: ID does not exist" containerID="4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"
Oct 30 21:45:18.087421 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.087341    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67"} err="failed to get container status \"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67\": rpc error: code = NotFound desc = could not find container \"4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67\": container with ID starting with 4265ea2476f81345041225113011b50f8b86a0d16abeb179543016c972b27e67 not found: ID does not exist"
Oct 30 21:45:18.087421 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.087358    4882 scope.go:122] "RemoveContainer" containerID="62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"
Oct 30 21:45:18.095778 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.095762    4882 scope.go:122] "RemoveContainer" containerID="62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"
Oct 30 21:45:18.096052 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:18.096035    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28\": container with ID starting with 62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28 not found: ID does not exist" containerID="62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"
Oct 30 21:45:18.096089 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.096064    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28"} err="failed to get container status \"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28\": rpc error: code = NotFound desc = could not find container \"62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28\": container with ID starting with 62475e98ec0197f8a47cdccef7b5125b707b52e4cb322bb5dd6be11206f31b28 not found: ID does not exist"
Oct 30 21:45:18.096089 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.096081    4882 scope.go:122] "RemoveContainer" containerID="168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"
Oct 30 21:45:18.104271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.104252    4882 scope.go:122] "RemoveContainer" containerID="168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"
Oct 30 21:45:18.104522 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:18.104503    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b\": container with ID starting with 168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b not found: ID does not exist" containerID="168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"
Oct 30 21:45:18.104606 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.104535    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b"} err="failed to get container status \"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b\": rpc error: code = NotFound desc = could not find container \"168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b\": container with ID starting with 168082b88f4819493feac937823706c9437440a8de945169ceff904d2952567b not found: ID does not exist"
Oct 30 21:45:18.243937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.243902    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:18.249973 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.249950    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6f6944bcdd-ljvv5"]
Oct 30 21:45:18.360285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.360250    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:45:18.364134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.364109    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-cp797"]
Oct 30 21:45:18.474253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.474167    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:18.478397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.478371    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6754bbb7d6-nf8jh"]
Oct 30 21:45:18.949713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.949675    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" path="/var/lib/kubelet/pods/a40a0c2c-6943-4572-a7f3-5a4388f54afe/volumes"
Oct 30 21:45:18.950045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.950033    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc33793-9b03-476e-836b-d801b9076037" path="/var/lib/kubelet/pods/edc33793-9b03-476e-836b-d801b9076037/volumes"
Oct 30 21:45:18.950323 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:18.950312    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" path="/var/lib/kubelet/pods/ef6bcba2-a205-4a03-8b97-6d60105ab6aa/volumes"
Oct 30 21:45:19.936361 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:19.936328    4882 scope.go:122] "RemoveContainer" containerID="28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b"
Oct 30 21:45:20.805660 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:20.805618    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:45:21.099850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:21.099818    4882 generic.go:356] "Generic (PLEG): container finished" podID="20538d23-6213-4385-ab16-74170d47532e" containerID="eaa3db42032dacdbd40c91c5c342d8613152f2d35fe9338072e1c46086724f29" exitCode=60
Oct 30 21:45:21.100273 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:21.099867    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" event={"ID":"20538d23-6213-4385-ab16-74170d47532e","Type":"ContainerDied","Data":"eaa3db42032dacdbd40c91c5c342d8613152f2d35fe9338072e1c46086724f29"}
Oct 30 21:45:21.100273 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:21.099920    4882 scope.go:122] "RemoveContainer" containerID="28a08872cf347a2c94ceb64da95e8eda70e185518de487962e9b5c859cc2013b"
Oct 30 21:45:21.100366 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:21.100283    4882 scope.go:122] "RemoveContainer" containerID="eaa3db42032dacdbd40c91c5c342d8613152f2d35fe9338072e1c46086724f29"
Oct 30 21:45:21.100555 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:21.100535    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889945-w87lt_odh-ai-gateway-infra(20538d23-6213-4385-ab16-74170d47532e)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" podUID="20538d23-6213-4385-ab16-74170d47532e"
Oct 30 21:45:22.235701 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.235672    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:22.372325 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.372300    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:22.535802 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.535716    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\") pod \"20538d23-6213-4385-ab16-74170d47532e\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") "
Oct 30 21:45:22.535802 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.535774    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\" (UniqueName: \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\") pod \"20538d23-6213-4385-ab16-74170d47532e\" (UID: \"20538d23-6213-4385-ab16-74170d47532e\") "
Oct 30 21:45:22.536132 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.536108    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca" pod "20538d23-6213-4385-ab16-74170d47532e" (UID: "20538d23-6213-4385-ab16-74170d47532e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:45:22.538099 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.538071    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t" pod "20538d23-6213-4385-ab16-74170d47532e" (UID: "20538d23-6213-4385-ab16-74170d47532e"). InnerVolumeSpecName "kube-api-access-7ft4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:45:22.637354 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.637316    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20538d23-6213-4385-ab16-74170d47532e-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:22.637354 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:22.637345    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7ft4t\" (UniqueName: \"kubernetes.io/projected/20538d23-6213-4385-ab16-74170d47532e-kube-api-access-7ft4t\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:45:23.112464 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:23.112434    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"
Oct 30 21:45:23.112646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:23.112461    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt" event={"ID":"20538d23-6213-4385-ab16-74170d47532e","Type":"ContainerDied","Data":"af29cde77a8d8f178e2dbd85584d6b02cd85319bc9c7a9d38412b08f9eb4ec6f"}
Oct 30 21:45:23.112646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:23.112525    4882 scope.go:122] "RemoveContainer" containerID="eaa3db42032dacdbd40c91c5c342d8613152f2d35fe9338072e1c46086724f29"
Oct 30 21:45:23.280443 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:23.280361    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:23.330760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:23.330713    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889945-w87lt"]
Oct 30 21:45:24.941259 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:24.941224    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20538d23-6213-4385-ab16-74170d47532e" path="/var/lib/kubelet/pods/20538d23-6213-4385-ab16-74170d47532e/volumes"
Oct 30 21:45:38.614628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.614537    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"]
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.614980    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" containerName="authorino"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.614992    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" containerName="authorino"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615008    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615014    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615021    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615026    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615036    4882 cpu_manager.go:414] "containerMap: removing container" podUID="edc33793-9b03-476e-836b-d801b9076037" containerName="authorino"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615041    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc33793-9b03-476e-836b-d801b9076037" containerName="authorino"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615064    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" containerName="authorino"
Oct 30 21:45:38.615069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615069    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" containerName="authorino"
Oct 30 21:45:38.615367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615146    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615154    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a40a0c2c-6943-4572-a7f3-5a4388f54afe" containerName="authorino"
Oct 30 21:45:38.615367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615161    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6bcba2-a205-4a03-8b97-6d60105ab6aa" containerName="authorino"
Oct 30 21:45:38.615367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615167    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:45:38.615367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.615173    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc33793-9b03-476e-836b-d801b9076037" containerName="authorino"
Oct 30 21:45:38.618265 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.618243    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.676085 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.676049    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"]
Oct 30 21:45:38.682317 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.682283    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jght\" (UniqueName: \"kubernetes.io/projected/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-kube-api-access-7jght\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.682476 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.682401    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.783327 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.783282    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.783526 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.783389    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jght\" (UniqueName: \"kubernetes.io/projected/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-kube-api-access-7jght\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.785762 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.785740    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-cert\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.846124 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.846095    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jght\" (UniqueName: \"kubernetes.io/projected/f7d5fad6-516c-4281-98f5-b8df1dc8c40e-kube-api-access-7jght\") pod \"llmisvc-controller-manager-6466c554d8-sdnsl\" (UID: \"f7d5fad6-516c-4281-98f5-b8df1dc8c40e\") " pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:38.930988 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:38.930890    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:39.300269 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:45:39.300235    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7d5fad6_516c_4281_98f5_b8df1dc8c40e.slice/crio-1f44783c0820e472ab1d1b748275a16d02d538fc8a420bc593f4ed891fc3c505 WatchSource:0}: Error finding container 1f44783c0820e472ab1d1b748275a16d02d538fc8a420bc593f4ed891fc3c505: Status 404 returned error can't find the container with id 1f44783c0820e472ab1d1b748275a16d02d538fc8a420bc593f4ed891fc3c505
Oct 30 21:45:39.302808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:39.302785    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"]
Oct 30 21:45:40.201660 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:40.201603    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl" event={"ID":"f7d5fad6-516c-4281-98f5-b8df1dc8c40e","Type":"ContainerStarted","Data":"410ca62859a204cf47fbc530529a8cb82d82a69c205a7a301134cb0204b45443"}
Oct 30 21:45:40.202125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:40.201674    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl" event={"ID":"f7d5fad6-516c-4281-98f5-b8df1dc8c40e","Type":"ContainerStarted","Data":"1f44783c0820e472ab1d1b748275a16d02d538fc8a420bc593f4ed891fc3c505"}
Oct 30 21:45:40.320924 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:40.320867    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl" podStartSLOduration=1.485248162 podStartE2EDuration="2.320852557s" totalImagesPullingTime="835.604395ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:45:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:45:40.318751178 +0000 UTC m=+463.492075605" watchObservedRunningTime="2026-10-30 21:45:40.320852557 +0000 UTC m=+463.494177026"
Oct 30 21:45:41.206728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:45:41.206700    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:45:50.806630 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:45:50.806566    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:46:12.220389 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.220353    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-6466c554d8-sdnsl"
Oct 30 21:46:12.393894 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.393839    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:46:12.394088 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.394065    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" podUID="72349f10-60de-4b70-a123-ef9f8f9d42ec" containerName="manager" containerID="cri-o://96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59" gracePeriod=30
Oct 30 21:46:12.645716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.645682    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:46:12.798631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.798531    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\" (UniqueName: \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\") pod \"72349f10-60de-4b70-a123-ef9f8f9d42ec\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") "
Oct 30 21:46:12.798784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.798636    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\" (UniqueName: \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\") pod \"72349f10-60de-4b70-a123-ef9f8f9d42ec\" (UID: \"72349f10-60de-4b70-a123-ef9f8f9d42ec\") "
Oct 30 21:46:12.800822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.800791    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t" pod "72349f10-60de-4b70-a123-ef9f8f9d42ec" (UID: "72349f10-60de-4b70-a123-ef9f8f9d42ec"). InnerVolumeSpecName "kube-api-access-ct46t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:46:12.800943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.800864    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert" pod "72349f10-60de-4b70-a123-ef9f8f9d42ec" (UID: "72349f10-60de-4b70-a123-ef9f8f9d42ec"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:46:12.899492 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.899454    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct46t\" (UniqueName: \"kubernetes.io/projected/72349f10-60de-4b70-a123-ef9f8f9d42ec-kube-api-access-ct46t\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:12.899492 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:12.899489    4882 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/72349f10-60de-4b70-a123-ef9f8f9d42ec-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:46:13.375994 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.375959    4882 generic.go:356] "Generic (PLEG): container finished" podID="72349f10-60de-4b70-a123-ef9f8f9d42ec" containerID="96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59" exitCode=0
Oct 30 21:46:13.376401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.376018    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"
Oct 30 21:46:13.376401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.376044    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" event={"ID":"72349f10-60de-4b70-a123-ef9f8f9d42ec","Type":"ContainerDied","Data":"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"}
Oct 30 21:46:13.376401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.376099    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq" event={"ID":"72349f10-60de-4b70-a123-ef9f8f9d42ec","Type":"ContainerDied","Data":"4b88bf928d0673ea95f78ddcc5cc681478ea9fb5ae12b897e6025f7bf686f369"}
Oct 30 21:46:13.376401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.376122    4882 scope.go:122] "RemoveContainer" containerID="96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"
Oct 30 21:46:13.384605 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.384568    4882 scope.go:122] "RemoveContainer" containerID="96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"
Oct 30 21:46:13.384895 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:46:13.384875    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59\": container with ID starting with 96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59 not found: ID does not exist" containerID="96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"
Oct 30 21:46:13.384967 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.384908    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59"} err="failed to get container status \"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59\": rpc error: code = NotFound desc = could not find container \"96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59\": container with ID starting with 96584a3703d55fa09a8b0afd5e6e8d585b00e5ee60dfc3fd52fbfe8b6eca0f59 not found: ID does not exist"
Oct 30 21:46:13.547213 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.547182    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:46:13.553646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:13.553622    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/llmisvc-controller-manager-869d5dcc5d-xtzhq"]
Oct 30 21:46:14.941494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:14.941459    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72349f10-60de-4b70-a123-ef9f8f9d42ec" path="/var/lib/kubelet/pods/72349f10-60de-4b70-a123-ef9f8f9d42ec/volumes"
Oct 30 21:46:20.806247 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:46:20.806206    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:46:50.805847 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:46:50.805724    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:46:55.376240 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:46:55.376207    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:47:05.604575 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:05.604540    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:47:18.409962 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:18.409852    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:47:20.805663 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:47:20.805626    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:47:33.946147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:33.946106    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:47:44.082323 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082287    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082728    4882 cpu_manager.go:414] "containerMap: removing container" podUID="72349f10-60de-4b70-a123-ef9f8f9d42ec" containerName="manager"
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082740    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72349f10-60de-4b70-a123-ef9f8f9d42ec" containerName="manager"
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082753    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082759    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082833    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="72349f10-60de-4b70-a123-ef9f8f9d42ec" containerName="manager"
Oct 30 21:47:44.082852 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.082842    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20538d23-6213-4385-ab16-74170d47532e" containerName="cleanup"
Oct 30 21:47:44.085799 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.085782    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.140943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.140911    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-dvk6k\""
Oct 30 21:47:44.191117 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.191075    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 21:47:44.220888 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.220855    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 21:47:44.238440 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238402    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238446    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238473    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238543    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmftp\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238701    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238741    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238777    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238913 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238895    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.238985 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.238973    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.339989 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.339903    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.339989 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.339964    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.339989 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.339988    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340007    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340023    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340082    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmftp\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340120    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340135    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340316 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340158    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340336    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340747 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340719    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340860    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.340934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.340882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.342523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.342501    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.342636 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.342609    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.408969 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.408925    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.409115 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.409005    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmftp\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksw98\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:44.698151 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:44.698065    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:45.125177 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:47:45.125115    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda92a87fb_b018_468f_bbd6_328c15452c94.slice/crio-9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673 WatchSource:0}: Error finding container 9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673: Status 404 returned error can't find the container with id 9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673
Oct 30 21:47:45.128271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:45.128245    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 21:47:45.833084 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:45.833046    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" event={"ID":"a92a87fb-b018-468f-bbd6-328c15452c94","Type":"ContainerStarted","Data":"9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673"}
Oct 30 21:47:49.161413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:49.161380    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:47:49.161713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:49.161454    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:47:49.161713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:49.161485    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:47:49.858219 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:49.858182    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" event={"ID":"a92a87fb-b018-468f-bbd6-328c15452c94","Type":"ContainerStarted","Data":"b23b4ef869dc6d8358628edff5bd04e7ab975c396185f9d3614ecff73098215b"}
Oct 30 21:47:50.699064 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:50.699006    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:50.700615 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:50.700571    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:50.700734 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:50.700641    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:50.805773 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:47:50.805735    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:47:51.699395 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:51.699359    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:51.699833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:51.699422    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:52.698469 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:52.698431    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:52.698685 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:52.698492    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:53.699254 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:53.699215    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:53.699823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:53.699282    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:54.698956 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:54.698921    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:54.699159 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:54.698972    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:55.699339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:55.699305    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:55.699764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:55.699358    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:56.699092 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:56.699044    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused" start-of-body=
Oct 30 21:47:56.699270 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:56.699112    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": dial tcp 10.132.0.43:15021: connect: connection refused"
Oct 30 21:47:56.991078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:56.990990    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:47:56.992290 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:56.992266    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:47:57.702816 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:57.702788    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:57.703084 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:57.703061    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:57.703832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:57.703812    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 21:47:57.877015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:47:57.876958    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podStartSLOduration=9.844372922 podStartE2EDuration="13.876943692s" totalImagesPullingTime="4.03257077s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:47:50.050054675 +0000 UTC m=+593.223379103" watchObservedRunningTime="2026-10-30 21:47:57.876943692 +0000 UTC m=+601.050268117"
Oct 30 21:48:03.162797 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.162757    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:48:03.163287 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.162954    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-6d6676b56-w5w5l" podUID="349015b9-cd29-44f3-b5fc-7360f1f5437b" containerName="authorino" containerID="cri-o://866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04" gracePeriod=30
Oct 30 21:48:03.421010 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.420950    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:48:03.447090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.447046    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\" (UniqueName: \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\") pod \"349015b9-cd29-44f3-b5fc-7360f1f5437b\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") "
Oct 30 21:48:03.447270 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.447137    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\" (UniqueName: \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\") pod \"349015b9-cd29-44f3-b5fc-7360f1f5437b\" (UID: \"349015b9-cd29-44f3-b5fc-7360f1f5437b\") "
Oct 30 21:48:03.449119 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.449084    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k" pod "349015b9-cd29-44f3-b5fc-7360f1f5437b" (UID: "349015b9-cd29-44f3-b5fc-7360f1f5437b"). InnerVolumeSpecName "kube-api-access-zvp2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:48:03.457827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.457799    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert" pod "349015b9-cd29-44f3-b5fc-7360f1f5437b" (UID: "349015b9-cd29-44f3-b5fc-7360f1f5437b"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:48:03.548860 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.548821    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zvp2k\" (UniqueName: \"kubernetes.io/projected/349015b9-cd29-44f3-b5fc-7360f1f5437b-kube-api-access-zvp2k\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:03.549014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.548887    4882 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/349015b9-cd29-44f3-b5fc-7360f1f5437b-tls-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:48:03.930215 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.930176    4882 generic.go:356] "Generic (PLEG): container finished" podID="349015b9-cd29-44f3-b5fc-7360f1f5437b" containerID="866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04" exitCode=0
Oct 30 21:48:03.930397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.930227    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d6676b56-w5w5l"
Oct 30 21:48:03.930397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.930245    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6676b56-w5w5l" event={"ID":"349015b9-cd29-44f3-b5fc-7360f1f5437b","Type":"ContainerDied","Data":"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"}
Oct 30 21:48:03.930397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.930275    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6d6676b56-w5w5l" event={"ID":"349015b9-cd29-44f3-b5fc-7360f1f5437b","Type":"ContainerDied","Data":"c5f7373e22175a0de2fddfbaf939386a0c9f20c165b6a49fbb0a8eab5b72607b"}
Oct 30 21:48:03.930397 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.930291    4882 scope.go:122] "RemoveContainer" containerID="866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"
Oct 30 21:48:03.939694 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.939675    4882 scope.go:122] "RemoveContainer" containerID="866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"
Oct 30 21:48:03.939995 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:48:03.939976    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04\": container with ID starting with 866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04 not found: ID does not exist" containerID="866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"
Oct 30 21:48:03.940074 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:03.940006    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04"} err="failed to get container status \"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04\": rpc error: code = NotFound desc = could not find container \"866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04\": container with ID starting with 866968eaae649cdfdfb1e3c4991a806b8a5f3612ad78df91ea0ef97375e31b04 not found: ID does not exist"
Oct 30 21:48:04.101571 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:04.101537    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:48:04.151923 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:04.151885    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d6676b56-w5w5l"]
Oct 30 21:48:04.940733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:48:04.940701    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="349015b9-cd29-44f3-b5fc-7360f1f5437b" path="/var/lib/kubelet/pods/349015b9-cd29-44f3-b5fc-7360f1f5437b/volumes"
Oct 30 21:48:20.806238 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:48:20.806193    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:48:50.806163 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:48:50.806073    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:49:20.805402 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:49:20.805365    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:49:24.744581 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:24.744544    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:49:29.241862 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.241820    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:49:29.242350 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.242268    4882 cpu_manager.go:414] "containerMap: removing container" podUID="349015b9-cd29-44f3-b5fc-7360f1f5437b" containerName="authorino"
Oct 30 21:49:29.242350 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.242280    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="349015b9-cd29-44f3-b5fc-7360f1f5437b" containerName="authorino"
Oct 30 21:49:29.242430 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.242360    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="349015b9-cd29-44f3-b5fc-7360f1f5437b" containerName="authorino"
Oct 30 21:49:29.245623 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.245606    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.300053 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.300023    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:49:29.347893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.347856    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-ffe7a1cf\""
Oct 30 21:49:29.347893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.347866    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-ffe7a1cf-dockercfg-22jxt\""
Oct 30 21:49:29.379861 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.379829    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.380056 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.379868    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk8k7\" (UniqueName: \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.481508 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.481468    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.481703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.481516    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk8k7\" (UniqueName: \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.482197 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.482175    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.544298 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.544227    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk8k7\" (UniqueName: \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\") pod \"payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.558633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.558602    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:29.927217 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:49:29.927186    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8c51c3_77c5_4cec_9061_41006696f332.slice/crio-65a52d92a8d4454dac3c2de76365218d9d680062c804a7ffb0058b1dd90d842e WatchSource:0}: Error finding container 65a52d92a8d4454dac3c2de76365218d9d680062c804a7ffb0058b1dd90d842e: Status 404 returned error can't find the container with id 65a52d92a8d4454dac3c2de76365218d9d680062c804a7ffb0058b1dd90d842e
Oct 30 21:49:29.929134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.929072    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:49:29.929668 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:29.929647    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 21:49:30.339605 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:30.339548    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" event={"ID":"4e8c51c3-77c5-4cec-9061-41006696f332","Type":"ContainerStarted","Data":"65a52d92a8d4454dac3c2de76365218d9d680062c804a7ffb0058b1dd90d842e"}
Oct 30 21:49:32.908995 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:32.908957    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:49:33.355950 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:33.355916    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" event={"ID":"4e8c51c3-77c5-4cec-9061-41006696f332","Type":"ContainerStarted","Data":"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"}
Oct 30 21:49:33.356169 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:33.356016    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:33.474505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:33.474444    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" podStartSLOduration=1.843185495 podStartE2EDuration="4.474427562s" totalImagesPullingTime="2.631242067s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:33.472236427 +0000 UTC m=+696.645560854" watchObservedRunningTime="2026-10-30 21:49:33.474427562 +0000 UTC m=+696.647752043"
Oct 30 21:49:35.689449 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.689413    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:49:35.693263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.693242    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.797343 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.797304    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-ffe7a1cf-serving-cert\""
Oct 30 21:49:35.797544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.797376    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Oct 30 21:49:35.797544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.797421    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-zr4d2\""
Oct 30 21:49:35.799656 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.799634    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:49:35.840433 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.840391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.840614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.840453    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.840614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.840598    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj45l\" (UniqueName: \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.840750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.840729    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.941413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.941317    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.941413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.941362    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.941413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.941388    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.941725 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.941449    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj45l\" (UniqueName: \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.942238 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.942213    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.943870 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.943838    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:35.943974 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:35.943889    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:36.003368 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:36.003338    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj45l\" (UniqueName: \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\") pod \"maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:36.006151 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:36.006116    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:36.377623 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:49:36.377575    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0df62bc_0f11_4797_8530_d9e50e70e9f1.slice/crio-7095d7737aa4ea7938e62e0495ad47d94156ee98967f3d80e5aa49d3e75ee3f1 WatchSource:0}: Error finding container 7095d7737aa4ea7938e62e0495ad47d94156ee98967f3d80e5aa49d3e75ee3f1: Status 404 returned error can't find the container with id 7095d7737aa4ea7938e62e0495ad47d94156ee98967f3d80e5aa49d3e75ee3f1
Oct 30 21:49:36.379218 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:36.379191    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:49:37.378240 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:37.378204    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerStarted","Data":"7095d7737aa4ea7938e62e0495ad47d94156ee98967f3d80e5aa49d3e75ee3f1"}
Oct 30 21:49:40.395989 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:40.395949    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerStarted","Data":"9f751bb2daa014fc2b76d5a25a3a022794c5f9e42f0a7f077926b217ef892936"}
Oct 30 21:49:40.396405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:40.396046    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:40.515288 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:40.515084    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" podStartSLOduration=2.503340573 podStartE2EDuration="5.515065975s" totalImagesPullingTime="3.011725402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:49:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:49:40.514122317 +0000 UTC m=+703.687446747" watchObservedRunningTime="2026-10-30 21:49:40.515065975 +0000 UTC m=+703.688390405"
Oct 30 21:49:44.365273 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:44.365245    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:49:45.441930 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:45.441865    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" podUID="845d50ae5087e4cd62e2c4035d1ecf3f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": context deadline exceeded"
Oct 30 21:49:46.402526 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:46.402475    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api" probeResult="failure" output="Get \"https://10.132.0.45:8443/health\": dial tcp 10.132.0.45:8443: connect: connection refused"
Oct 30 21:49:49.433148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:49.433113    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:49:50.445427 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:50.445392    4882 generic.go:356] "Generic (PLEG): container finished" podID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerID="9f751bb2daa014fc2b76d5a25a3a022794c5f9e42f0a7f077926b217ef892936" exitCode=1
Oct 30 21:49:50.445861 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:50.445469    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerDied","Data":"9f751bb2daa014fc2b76d5a25a3a022794c5f9e42f0a7f077926b217ef892936"}
Oct 30 21:49:50.445921 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:50.445910    4882 scope.go:122] "RemoveContainer" containerID="9f751bb2daa014fc2b76d5a25a3a022794c5f9e42f0a7f077926b217ef892936"
Oct 30 21:49:50.807533 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:49:50.807445    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:49:51.452495 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:51.452463    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerStarted","Data":"d7848b3a8f04675496845d566f443a3093ff181e25f1a247a4a338397053684d"}
Oct 30 21:49:51.452898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:51.452740    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:49:57.462325 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:49:57.462295    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:50:16.497171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:16.497091    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:50:20.805119 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:50:20.805083    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:50:31.387201 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:31.387167    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:50:36.387873 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.387813    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" podUID="4e8c51c3-77c5-4cec-9061-41006696f332" containerName="payload-pre-processing" containerID="cri-o://e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f" gracePeriod=25
Oct 30 21:50:36.645577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.645514    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:50:36.674881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.674849    4882 generic.go:356] "Generic (PLEG): container finished" podID="4e8c51c3-77c5-4cec-9061-41006696f332" containerID="e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f" exitCode=0
Oct 30 21:50:36.675058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.674885    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" event={"ID":"4e8c51c3-77c5-4cec-9061-41006696f332","Type":"ContainerDied","Data":"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"}
Oct 30 21:50:36.675058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.674950    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p" event={"ID":"4e8c51c3-77c5-4cec-9061-41006696f332","Type":"ContainerDied","Data":"65a52d92a8d4454dac3c2de76365218d9d680062c804a7ffb0058b1dd90d842e"}
Oct 30 21:50:36.675058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.674950    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"
Oct 30 21:50:36.675058 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.674974    4882 scope.go:122] "RemoveContainer" containerID="e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"
Oct 30 21:50:36.685337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.685315    4882 scope.go:122] "RemoveContainer" containerID="e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"
Oct 30 21:50:36.685774 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:50:36.685742    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f\": container with ID starting with e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f not found: ID does not exist" containerID="e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"
Oct 30 21:50:36.685926 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.685782    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f"} err="failed to get container status \"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f\": rpc error: code = NotFound desc = could not find container \"e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f\": container with ID starting with e38598ddeaa4f118ffdbbbe19013f3af20172474781a9817f5cf80fa773dd31f not found: ID does not exist"
Oct 30 21:50:36.726778 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.726743    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\" (UniqueName: \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\") pod \"4e8c51c3-77c5-4cec-9061-41006696f332\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") "
Oct 30 21:50:36.726943 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.726792    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\") pod \"4e8c51c3-77c5-4cec-9061-41006696f332\" (UID: \"4e8c51c3-77c5-4cec-9061-41006696f332\") "
Oct 30 21:50:36.727217 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.727189    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume" pod "4e8c51c3-77c5-4cec-9061-41006696f332" (UID: "4e8c51c3-77c5-4cec-9061-41006696f332"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:36.729007 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.728978    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7" pod "4e8c51c3-77c5-4cec-9061-41006696f332" (UID: "4e8c51c3-77c5-4cec-9061-41006696f332"). InnerVolumeSpecName "kube-api-access-hk8k7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:36.827436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.827401    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk8k7\" (UniqueName: \"kubernetes.io/projected/4e8c51c3-77c5-4cec-9061-41006696f332-kube-api-access-hk8k7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:36.827436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:36.827431    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e8c51c3-77c5-4cec-9061-41006696f332-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:37.092184 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:37.092143    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:50:37.096354 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:37.096327    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-ffe7a1cf-5d5f9fd695-6qx4p"]
Oct 30 21:50:38.475744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.475713    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:50:38.476171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.475912    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api" containerID="cri-o://d7848b3a8f04675496845d566f443a3093ff181e25f1a247a4a338397053684d" gracePeriod=30
Oct 30 21:50:38.688707 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.688669    4882 generic.go:356] "Generic (PLEG): container finished" podID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerID="d7848b3a8f04675496845d566f443a3093ff181e25f1a247a4a338397053684d" exitCode=0
Oct 30 21:50:38.688899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.688710    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerDied","Data":"d7848b3a8f04675496845d566f443a3093ff181e25f1a247a4a338397053684d"}
Oct 30 21:50:38.688899 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.688787    4882 scope.go:122] "RemoveContainer" containerID="9f751bb2daa014fc2b76d5a25a3a022794c5f9e42f0a7f077926b217ef892936"
Oct 30 21:50:38.740662 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.740581    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:50:38.746625 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.746604    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\" (UniqueName: \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\") pod \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") "
Oct 30 21:50:38.746766 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.746676    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\") pod \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") "
Oct 30 21:50:38.746766 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.746739    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\") pod \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") "
Oct 30 21:50:38.746885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.746780    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\") pod \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\" (UID: \"d0df62bc-0f11-4797-8530-d9e50e70e9f1\") "
Oct 30 21:50:38.747382 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.747351    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca" pod "d0df62bc-0f11-4797-8530-d9e50e70e9f1" (UID: "d0df62bc-0f11-4797-8530-d9e50e70e9f1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:50:38.748998 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.748967    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls" pod "d0df62bc-0f11-4797-8530-d9e50e70e9f1" (UID: "d0df62bc-0f11-4797-8530-d9e50e70e9f1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:38.749250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.749229    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l" pod "d0df62bc-0f11-4797-8530-d9e50e70e9f1" (UID: "d0df62bc-0f11-4797-8530-d9e50e70e9f1"). InnerVolumeSpecName "kube-api-access-hj45l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:50:38.749329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.749267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs" pod "d0df62bc-0f11-4797-8530-d9e50e70e9f1" (UID: "d0df62bc-0f11-4797-8530-d9e50e70e9f1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:50:38.847732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.847698    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj45l\" (UniqueName: \"kubernetes.io/projected/d0df62bc-0f11-4797-8530-d9e50e70e9f1-kube-api-access-hj45l\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:38.847732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.847727    4882 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-metrics-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:38.847732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.847736    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0df62bc-0f11-4797-8530-d9e50e70e9f1-maas-api-tls\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:38.847974 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.847747    4882 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0df62bc-0f11-4797-8530-d9e50e70e9f1-openshift-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:50:38.941944 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:38.941911    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4e8c51c3-77c5-4cec-9061-41006696f332" path="/var/lib/kubelet/pods/4e8c51c3-77c5-4cec-9061-41006696f332/volumes"
Oct 30 21:50:39.694802 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:39.694767    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5" event={"ID":"d0df62bc-0f11-4797-8530-d9e50e70e9f1","Type":"ContainerDied","Data":"7095d7737aa4ea7938e62e0495ad47d94156ee98967f3d80e5aa49d3e75ee3f1"}
Oct 30 21:50:39.695246 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:39.694777    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"
Oct 30 21:50:39.695246 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:39.694860    4882 scope.go:122] "RemoveContainer" containerID="d7848b3a8f04675496845d566f443a3093ff181e25f1a247a4a338397053684d"
Oct 30 21:50:39.865336 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:39.865304    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:50:39.914523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:39.914488    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-ffe7a1cf-c8b6448d7-v29v5"]
Oct 30 21:50:40.941907 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:40.941847    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" path="/var/lib/kubelet/pods/d0df62bc-0f11-4797-8530-d9e50e70e9f1/volumes"
Oct 30 21:50:42.133124 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:42.133086    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:50:49.867253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:49.867212    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:50:50.805138 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:50:50.805100    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:50:58.335864 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:50:58.335833    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:51:05.707030 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:05.706992    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:51:13.382552 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.382517    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.382945    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.382958    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.382978    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4e8c51c3-77c5-4cec-9061-41006696f332" containerName="payload-pre-processing"
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.382983    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4e8c51c3-77c5-4cec-9061-41006696f332" containerName="payload-pre-processing"
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.383055    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:13.383143 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.383065    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4e8c51c3-77c5-4cec-9061-41006696f332" containerName="payload-pre-processing"
Oct 30 21:51:13.385815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.385799    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.475289 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.475254    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"openshift-service-ca.crt\""
Oct 30 21:51:13.475466 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.475313    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"kube-root-ca.crt\""
Oct 30 21:51:13.475466 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.475313    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"simulator-w0-9a5623-kserve-self-signed-certs\""
Oct 30 21:51:13.481476 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.481456    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"default-dockercfg-45vff\""
Oct 30 21:51:13.483525 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483504    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.483633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.483705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483688    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.483760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483746    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.483803 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483792    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.483836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.483822    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjmdv\" (UniqueName: \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.491373 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.491349    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 21:51:13.584420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584378    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjmdv\" (UniqueName: \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.584635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584450    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.584635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584504    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.584635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584541    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.584808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584658    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.584808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584763    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.585025 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.584994    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.585025 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.585010    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.585209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.585135    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.587504 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.587478    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.587695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.587677    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.646922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.646845    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjmdv\" (UniqueName: \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\") pod \"simulator-w0-9a5623-kserve-856b659b97-k6s7k\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") " pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:13.697875 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:13.697807    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:14.080150 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:51:14.080106    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1bdee079_40db_4af7_8b85_0bde190d3523.slice/crio-33057e4d812b0b85aca3d7cc1285c034568fb77b0dadceb16884066adce180d2 WatchSource:0}: Error finding container 33057e4d812b0b85aca3d7cc1285c034568fb77b0dadceb16884066adce180d2: Status 404 returned error can't find the container with id 33057e4d812b0b85aca3d7cc1285c034568fb77b0dadceb16884066adce180d2
Oct 30 21:51:14.082057 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:14.082033    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 21:51:14.871677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:14.871642    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" event={"ID":"1bdee079-40db-4af7-8b85-0bde190d3523","Type":"ContainerStarted","Data":"33057e4d812b0b85aca3d7cc1285c034568fb77b0dadceb16884066adce180d2"}
Oct 30 21:51:15.877693 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:15.877655    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" event={"ID":"1bdee079-40db-4af7-8b85-0bde190d3523","Type":"ContainerStarted","Data":"c0a9df4741d406b358c4bf62347782bc59fc7624d3dfb69bac47093257ede46d"}
Oct 30 21:51:16.001178 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:16.001074    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podStartSLOduration=1.446605485 podStartE2EDuration="3.001058601s" totalImagesPullingTime="1.554453116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:15.998946283 +0000 UTC m=+799.172270710" watchObservedRunningTime="2026-10-30 21:51:16.001058601 +0000 UTC m=+799.174383036"
Oct 30 21:51:16.698216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:16.698177    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:17.141502 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:17.141468    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:51:20.808364 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:51:20.808319    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:51:23.698386 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:23.698345    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:23.718287 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:23.718252    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:23.926450 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:23.926419    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:51:34.333044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:34.333011    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:51:39.682403 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.682311    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 21:51:39.683002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.682979    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:39.683100 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.683006    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:39.683152 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.683120    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d0df62bc-0f11-4797-8530-d9e50e70e9f1" containerName="maas-api"
Oct 30 21:51:39.686229 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.686206    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730236 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730201    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730252    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730401 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730375    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730408    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730443    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkjn7\" (UniqueName: \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.730536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.730508    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.790148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.790116    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"default-dockercfg-c758n\""
Oct 30 21:51:39.790148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.790136    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"simulator-w4-26743a-kserve-self-signed-certs\""
Oct 30 21:51:39.790148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.790153    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"kube-root-ca.crt\""
Oct 30 21:51:39.790357 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.790162    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-26743a\"/\"openshift-service-ca.crt\""
Oct 30 21:51:39.792379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.792359    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 21:51:39.831825 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831790    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.831992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831833    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.831992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831852    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.831992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831881    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkjn7\" (UniqueName: \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.831992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831912    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.831992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.831977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.832249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.832224    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.832328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.832311    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.832405 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.832384    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.834157 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.834124    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.834417 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.834396    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:39.893790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:39.893760    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkjn7\" (UniqueName: \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\") pod \"simulator-w4-26743a-kserve-69759cd7c8-fk4qc\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") " pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:40.030870 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:40.030779    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:40.403863 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:51:40.403818    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08ee8652_249f_4705_b037_6be8f460e4ab.slice/crio-149e88a0f31e2e042f5720cfcc3c3ae6ef1de01ce5e66ec96f724010ddfca023 WatchSource:0}: Error finding container 149e88a0f31e2e042f5720cfcc3c3ae6ef1de01ce5e66ec96f724010ddfca023: Status 404 returned error can't find the container with id 149e88a0f31e2e042f5720cfcc3c3ae6ef1de01ce5e66ec96f724010ddfca023
Oct 30 21:51:40.405577 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:40.405529    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 21:51:41.000009 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:40.999972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" event={"ID":"08ee8652-249f-4705-b037-6be8f460e4ab","Type":"ContainerStarted","Data":"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"}
Oct 30 21:51:41.000009 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:41.000015    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" event={"ID":"08ee8652-249f-4705-b037-6be8f460e4ab","Type":"ContainerStarted","Data":"149e88a0f31e2e042f5720cfcc3c3ae6ef1de01ce5e66ec96f724010ddfca023"}
Oct 30 21:51:41.031880 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:41.031844    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:41.130619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:41.130545    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podStartSLOduration=2.130531853 podStartE2EDuration="2.130531853s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:51:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:51:41.129295099 +0000 UTC m=+824.302619524" watchObservedRunningTime="2026-10-30 21:51:41.130531853 +0000 UTC m=+824.303856279"
Oct 30 21:51:44.416481 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:44.416443    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:51:50.031126 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:50.031081    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:50.045245 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:50.045213    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:50.057336 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:51:50.057313    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:51:50.804893 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:51:50.804845    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:52:09.409817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:09.409780    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:52:13.003977 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.003934    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 21:52:13.007680 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.007657    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.033169 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.033134    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 21:52:13.052136 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052106    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.052136 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052140    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.052474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052291    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.052474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052350    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqvtm\" (UniqueName: \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.052474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052421    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.052686 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.052547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.061044 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.061020    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-9a5623\"/\"premium-w0-9a5623-kserve-self-signed-certs\""
Oct 30 21:52:13.153659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153626    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.153659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153664    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.153932 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153688    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.153932 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153724    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqvtm\" (UniqueName: \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.153932 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153747    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.153932 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.153777    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.154168 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.154091    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.154168 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.154141    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.154271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.154189    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.156104 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.156077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.156402 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.156381    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.216306 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.216275    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqvtm\" (UniqueName: \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\") pod \"premium-w0-9a5623-kserve-78b4b7457b-7rrqt\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") " pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.323050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.322957    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:13.697820 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:52:13.697784    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc62c3cf7_5a2c_4c05_987c_4c2cf8188928.slice/crio-d46f27ae1f8136a018973e1e9f419240f39e60c37781b6387ab34a2d3f802f42 WatchSource:0}: Error finding container d46f27ae1f8136a018973e1e9f419240f39e60c37781b6387ab34a2d3f802f42: Status 404 returned error can't find the container with id d46f27ae1f8136a018973e1e9f419240f39e60c37781b6387ab34a2d3f802f42
Oct 30 21:52:13.698699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:13.698670    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 21:52:14.167439 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:14.167394    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" event={"ID":"c62c3cf7-5a2c-4c05-987c-4c2cf8188928","Type":"ContainerStarted","Data":"1dba39d3023d4c7d15661ec3251d091703d1e8cb1de86e56069428d90feb3bcb"}
Oct 30 21:52:14.167927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:14.167446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" event={"ID":"c62c3cf7-5a2c-4c05-987c-4c2cf8188928","Type":"ContainerStarted","Data":"d46f27ae1f8136a018973e1e9f419240f39e60c37781b6387ab34a2d3f802f42"}
Oct 30 21:52:14.286332 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:14.286281    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podStartSLOduration=2.286263785 podStartE2EDuration="2.286263785s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:14.284704638 +0000 UTC m=+857.458029066" watchObservedRunningTime="2026-10-30 21:52:14.286263785 +0000 UTC m=+857.459588211"
Oct 30 21:52:14.323465 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:14.323430    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:20.804974 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:52:20.804938    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:52:23.328610 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:23.325053    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:23.344336 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:23.344304    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:24.227050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:24.227024    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:52:28.733622 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:28.733573    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:52:38.972454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:38.972417    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 21:52:38.976357 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:38.976335    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.009710 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.009676    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.009881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.009727    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.009881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.009833    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.009881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.009874    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.010009 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.009967    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.010046 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.010010    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.080114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.080064    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"openshift-service-ca.crt\""
Oct 30 21:52:39.080114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.080068    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"kube-root-ca.crt\""
Oct 30 21:52:39.080369 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.080162    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"default-dockercfg-7bt4g\""
Oct 30 21:52:39.080369 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.080195    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"simulator-w1-3c4802-kserve-self-signed-certs\""
Oct 30 21:52:39.082311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.082291    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 21:52:39.110671 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110639    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.110798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110676    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.110798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110711    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.110798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.110951 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110904    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.111006 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.110947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.111121 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.111082    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.111203 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.111180    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.111262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.111210    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.113025 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.113000    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.113750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.113729    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.185510 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.185476    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\") pod \"simulator-w1-3c4802-kserve-75bcf5d784-l7szx\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") " pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.289113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.289030    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:39.665296 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:52:39.665259    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod37a3fbe2_9e70_49d9_a40c_c4b22aa90304.slice/crio-a63662a0b099c3efc4c2ebcd510fec671f8c508315684bb11d1205ca5fcf7871 WatchSource:0}: Error finding container a63662a0b099c3efc4c2ebcd510fec671f8c508315684bb11d1205ca5fcf7871: Status 404 returned error can't find the container with id a63662a0b099c3efc4c2ebcd510fec671f8c508315684bb11d1205ca5fcf7871
Oct 30 21:52:39.667335 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:39.667312    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 21:52:40.294416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:40.294379    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" event={"ID":"37a3fbe2-9e70-49d9-a40c-c4b22aa90304","Type":"ContainerStarted","Data":"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"}
Oct 30 21:52:40.294980 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:40.294428    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" event={"ID":"37a3fbe2-9e70-49d9-a40c-c4b22aa90304","Type":"ContainerStarted","Data":"a63662a0b099c3efc4c2ebcd510fec671f8c508315684bb11d1205ca5fcf7871"}
Oct 30 21:52:40.413514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:40.413456    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podStartSLOduration=2.413441388 podStartE2EDuration="2.413441388s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:52:40.412882623 +0000 UTC m=+883.586207050" watchObservedRunningTime="2026-10-30 21:52:40.413441388 +0000 UTC m=+883.586765816"
Oct 30 21:52:41.289147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:41.289102    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:49.094571 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:49.094524    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:52:49.289302 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:49.289269    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:49.302036 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:49.302004    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:49.351379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:49.351303    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:52:50.806360 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:52:50.806319    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:52:57.029217 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:57.029189    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:52:57.030524 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:57.030501    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:52:59.449380 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.449343    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 21:52:59.453419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.453398    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.509371 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.509335    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 21:52:59.511460 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511431    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.511643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511492    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.511643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511576    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.511780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511657    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.511780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511698    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8wtt\" (UniqueName: \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.511883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.511801    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.557111 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.557078    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"openshift-service-ca.crt\""
Oct 30 21:52:59.557111 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.557101    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"kube-root-ca.crt\""
Oct 30 21:52:59.557379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.557112    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"premium-w2-56f3fe-kserve-self-signed-certs\""
Oct 30 21:52:59.557379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.557353    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"default-dockercfg-rz8s4\""
Oct 30 21:52:59.612520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.612520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612532    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8wtt\" (UniqueName: \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.612785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.612785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612612    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.612785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612654    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.612785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612684    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.613021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.612985    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.613021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.613002    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.613089 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.613044    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.615079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.615055    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.615238 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.615222    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.675297 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.675254    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8wtt\" (UniqueName: \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\") pod \"premium-w2-56f3fe-kserve-769d49fff-469xt\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") " pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:52:59.767246 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:52:59.767153    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:53:00.147987 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:53:00.147953    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28c383d2_e556_40d6_a361_2291d3e2613d.slice/crio-b7af354acabe4719b10ea59ea01158393f2fa9fc7ab501a242d9c883940d61da WatchSource:0}: Error finding container b7af354acabe4719b10ea59ea01158393f2fa9fc7ab501a242d9c883940d61da: Status 404 returned error can't find the container with id b7af354acabe4719b10ea59ea01158393f2fa9fc7ab501a242d9c883940d61da
Oct 30 21:53:00.150091 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:00.150059    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 21:53:00.393290 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:00.393251    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" event={"ID":"28c383d2-e556-40d6-a361-2291d3e2613d","Type":"ContainerStarted","Data":"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"}
Oct 30 21:53:00.393464 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:00.393299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" event={"ID":"28c383d2-e556-40d6-a361-2291d3e2613d","Type":"ContainerStarted","Data":"b7af354acabe4719b10ea59ea01158393f2fa9fc7ab501a242d9c883940d61da"}
Oct 30 21:53:00.514842 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:00.514721    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podStartSLOduration=1.514704437 podStartE2EDuration="1.514704437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:52:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:00.512202136 +0000 UTC m=+903.685526566" watchObservedRunningTime="2026-10-30 21:53:00.514704437 +0000 UTC m=+903.688028864"
Oct 30 21:53:00.768123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:00.768041    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:53:09.768137 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:09.768050    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:53:09.780744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:09.780716    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:53:10.454221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:10.454190    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:53:20.811767 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:53:20.811728    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:53:22.779909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:22.779865    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:53:41.121604 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.121562    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"]
Oct 30 21:53:41.125601 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.125560    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.180528 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.180496    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"]
Oct 30 21:53:41.225201 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.225160    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.225494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.225245    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.225494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.225377    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9fbrs\" (UniqueName: \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.227916    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:41.228159 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.228145    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:53:41.326140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.326102    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9fbrs\" (UniqueName: \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.326329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.326171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.326329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.326219    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.326879 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.326855    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.328716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.328699    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.388305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.388231    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9fbrs\" (UniqueName: \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\") pod \"maas-api-revoke-sub-95b299a2-f8wgw\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.439955 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.439916    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:41.809035 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:53:41.808992    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod275063ec_f8d8_4fb9_9828_1ccdfb239905.slice/crio-31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb WatchSource:0}: Error finding container 31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb: Status 404 returned error can't find the container with id 31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb
Oct 30 21:53:41.810856 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:41.810831    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"]
Oct 30 21:53:42.602709 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:42.602670    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw" event={"ID":"275063ec-f8d8-4fb9-9828-1ccdfb239905","Type":"ContainerStarted","Data":"d561546720c96389768bacef32d50e8cabc9ad0fe74678f71304a145cd80e7ba"}
Oct 30 21:53:42.603129 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:42.602726    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw" event={"ID":"275063ec-f8d8-4fb9-9828-1ccdfb239905","Type":"ContainerStarted","Data":"31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb"}
Oct 30 21:53:42.726324 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:42.726271    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw" podStartSLOduration=1.726256175 podStartE2EDuration="1.726256175s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:53:42.725056849 +0000 UTC m=+945.898381275" watchObservedRunningTime="2026-10-30 21:53:42.726256175 +0000 UTC m=+945.899580601"
Oct 30 21:53:43.609440 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:43.609405    4882 generic.go:356] "Generic (PLEG): container finished" podID="275063ec-f8d8-4fb9-9828-1ccdfb239905" containerID="d561546720c96389768bacef32d50e8cabc9ad0fe74678f71304a145cd80e7ba" exitCode=0
Oct 30 21:53:43.609896 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:43.609494    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw" event={"ID":"275063ec-f8d8-4fb9-9828-1ccdfb239905","Type":"ContainerDied","Data":"d561546720c96389768bacef32d50e8cabc9ad0fe74678f71304a145cd80e7ba"}
Oct 30 21:53:44.752351 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.752321    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:44.862192 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.862159    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\") pod \"275063ec-f8d8-4fb9-9828-1ccdfb239905\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") "
Oct 30 21:53:44.862363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.862208    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\" (UniqueName: \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\") pod \"275063ec-f8d8-4fb9-9828-1ccdfb239905\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") "
Oct 30 21:53:44.862363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.862264    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\") pod \"275063ec-f8d8-4fb9-9828-1ccdfb239905\" (UID: \"275063ec-f8d8-4fb9-9828-1ccdfb239905\") "
Oct 30 21:53:44.862551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.862527    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca" pod "275063ec-f8d8-4fb9-9828-1ccdfb239905" (UID: "275063ec-f8d8-4fb9-9828-1ccdfb239905"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:53:44.864413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.864385    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca" pod "275063ec-f8d8-4fb9-9828-1ccdfb239905" (UID: "275063ec-f8d8-4fb9-9828-1ccdfb239905"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:53:44.864559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.864451    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs" pod "275063ec-f8d8-4fb9-9828-1ccdfb239905" (UID: "275063ec-f8d8-4fb9-9828-1ccdfb239905"). InnerVolumeSpecName "kube-api-access-9fbrs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:53:44.963648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.963618    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:44.963648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.963646    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/275063ec-f8d8-4fb9-9828-1ccdfb239905-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:44.963836 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:44.963658    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9fbrs\" (UniqueName: \"kubernetes.io/projected/275063ec-f8d8-4fb9-9828-1ccdfb239905-kube-api-access-9fbrs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:53:45.620651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:45.620623    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"
Oct 30 21:53:45.620651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:45.620618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw" event={"ID":"275063ec-f8d8-4fb9-9828-1ccdfb239905","Type":"ContainerDied","Data":"31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb"}
Oct 30 21:53:45.620872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:45.620681    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31492c3c47f03696439dd1a590107ed62cc33a91b6c7e6ca4eed9ddd6055f2fb"
Oct 30 21:53:46.755493 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:46.755457    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:53:50.806660 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:53:50.806616    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:53:59.042823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.042788    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"]
Oct 30 21:53:59.043261 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.043227    4882 cpu_manager.go:414] "containerMap: removing container" podUID="275063ec-f8d8-4fb9-9828-1ccdfb239905" containerName="revoke-keys"
Oct 30 21:53:59.043261 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.043239    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="275063ec-f8d8-4fb9-9828-1ccdfb239905" containerName="revoke-keys"
Oct 30 21:53:59.043349 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.043337    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="275063ec-f8d8-4fb9-9828-1ccdfb239905" containerName="revoke-keys"
Oct 30 21:53:59.046881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.046860    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.102835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.102791    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:53:59.153435 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.153399    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:53:59.155687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.155659    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"]
Oct 30 21:53:59.202502 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.202460    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.202775 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.202751    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-795db\" (UniqueName: \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.202834 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.202792    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.304022 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.303941    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-795db\" (UniqueName: \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.304022 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.303979    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.304221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.304041    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.304688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.304665    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.306378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.306353    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.366194 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.366162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-795db\" (UniqueName: \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\") pod \"maas-api-revoke-sub-5c9e9eb3-k7vc4\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:53:59.661147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:53:59.661112    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:54:00.036450 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:00.036324    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod644a7b86_0693_46b5_bfa5_6f370b334013.slice/crio-a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98 WatchSource:0}: Error finding container a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98: Status 404 returned error can't find the container with id a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98
Oct 30 21:54:00.039041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:00.038900    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"]
Oct 30 21:54:00.698461 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:00.698366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4" event={"ID":"644a7b86-0693-46b5-bfa5-6f370b334013","Type":"ContainerStarted","Data":"3ca38c406be1ffca38848142b5100ed4c86e7961de90cf7a5afa258049015ccc"}
Oct 30 21:54:00.698987 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:00.698482    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4" event={"ID":"644a7b86-0693-46b5-bfa5-6f370b334013","Type":"ContainerStarted","Data":"a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98"}
Oct 30 21:54:00.823417 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:00.823356    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4" podStartSLOduration=2.8233422729999997 podStartE2EDuration="2.823342273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:53:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:00.821842215 +0000 UTC m=+963.995166641" watchObservedRunningTime="2026-10-30 21:54:00.823342273 +0000 UTC m=+963.996666699"
Oct 30 21:54:01.704270 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:01.704233    4882 generic.go:356] "Generic (PLEG): container finished" podID="644a7b86-0693-46b5-bfa5-6f370b334013" containerID="3ca38c406be1ffca38848142b5100ed4c86e7961de90cf7a5afa258049015ccc" exitCode=0
Oct 30 21:54:01.704270 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:01.704275    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4" event={"ID":"644a7b86-0693-46b5-bfa5-6f370b334013","Type":"ContainerDied","Data":"3ca38c406be1ffca38848142b5100ed4c86e7961de90cf7a5afa258049015ccc"}
Oct 30 21:54:02.849638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.849611    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:54:02.941806 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.941779    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\") pod \"644a7b86-0693-46b5-bfa5-6f370b334013\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") "
Oct 30 21:54:02.941966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.941870    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\") pod \"644a7b86-0693-46b5-bfa5-6f370b334013\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") "
Oct 30 21:54:02.941966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.941926    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\" (UniqueName: \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\") pod \"644a7b86-0693-46b5-bfa5-6f370b334013\" (UID: \"644a7b86-0693-46b5-bfa5-6f370b334013\") "
Oct 30 21:54:02.942207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.942179    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca" pod "644a7b86-0693-46b5-bfa5-6f370b334013" (UID: "644a7b86-0693-46b5-bfa5-6f370b334013"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:02.944082 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.944055    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db" pod "644a7b86-0693-46b5-bfa5-6f370b334013" (UID: "644a7b86-0693-46b5-bfa5-6f370b334013"). InnerVolumeSpecName "kube-api-access-795db". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:02.944082 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:02.944058    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca" pod "644a7b86-0693-46b5-bfa5-6f370b334013" (UID: "644a7b86-0693-46b5-bfa5-6f370b334013"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:03.042960 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.042868    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:03.042960 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.042903    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-795db\" (UniqueName: \"kubernetes.io/projected/644a7b86-0693-46b5-bfa5-6f370b334013-kube-api-access-795db\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:03.042960 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.042920    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/644a7b86-0693-46b5-bfa5-6f370b334013-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:03.714983 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.714954    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"
Oct 30 21:54:03.714983 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.714972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4" event={"ID":"644a7b86-0693-46b5-bfa5-6f370b334013","Type":"ContainerDied","Data":"a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98"}
Oct 30 21:54:03.715211 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:03.715016    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a31817e6ea6acee17bf6cbccee5d5a7c888be7ee6c0c8527abbc154ae028dc98"
Oct 30 21:54:11.809188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.809150    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"]
Oct 30 21:54:11.809628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.809614    4882 cpu_manager.go:414] "containerMap: removing container" podUID="644a7b86-0693-46b5-bfa5-6f370b334013" containerName="revoke-keys"
Oct 30 21:54:11.809672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.809628    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="644a7b86-0693-46b5-bfa5-6f370b334013" containerName="revoke-keys"
Oct 30 21:54:11.809724 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.809715    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="644a7b86-0693-46b5-bfa5-6f370b334013" containerName="revoke-keys"
Oct 30 21:54:11.813102 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.813084    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:11.925990 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.925953    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:54:11.928352 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.928319    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"]
Oct 30 21:54:11.933598 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.933558    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:11.933675 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.933627    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwsm9\" (UniqueName: \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:11.933750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.933731    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:11.970547 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:11.970518    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:12.035146 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.035112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwsm9\" (UniqueName: \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.035396 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.035176    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.035396 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.035252    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.035917 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.035876    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.037759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.037735    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.097150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.097115    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwsm9\" (UniqueName: \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\") pod \"maas-api-revoke-sub-55162fd0-hgpsl\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.126120 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.126088    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:12.499846 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:12.499804    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b77d4cb_5b07_448d_952a_945ca8ebf299.slice/crio-bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115 WatchSource:0}: Error finding container bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115: Status 404 returned error can't find the container with id bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115
Oct 30 21:54:12.501523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.501494    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"]
Oct 30 21:54:12.762911 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:12.762723    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl" event={"ID":"0b77d4cb-5b07-448d-952a-945ca8ebf299","Type":"ContainerStarted","Data":"bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115"}
Oct 30 21:54:13.769274 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:13.769191    4882 generic.go:356] "Generic (PLEG): container finished" podID="0b77d4cb-5b07-448d-952a-945ca8ebf299" containerID="be341226a7052dd07cc251819f062fa1fff35142736d9b77d436c4a2c929df0d" exitCode=0
Oct 30 21:54:13.769274 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:13.769244    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl" event={"ID":"0b77d4cb-5b07-448d-952a-945ca8ebf299","Type":"ContainerDied","Data":"be341226a7052dd07cc251819f062fa1fff35142736d9b77d436c4a2c929df0d"}
Oct 30 21:54:14.918247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.918215    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:14.967976 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.967942    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\" (UniqueName: \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\") pod \"0b77d4cb-5b07-448d-952a-945ca8ebf299\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") "
Oct 30 21:54:14.968177 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.968036    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\") pod \"0b77d4cb-5b07-448d-952a-945ca8ebf299\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") "
Oct 30 21:54:14.968177 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.968146    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\") pod \"0b77d4cb-5b07-448d-952a-945ca8ebf299\" (UID: \"0b77d4cb-5b07-448d-952a-945ca8ebf299\") "
Oct 30 21:54:14.968458 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.968431    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca" pod "0b77d4cb-5b07-448d-952a-945ca8ebf299" (UID: "0b77d4cb-5b07-448d-952a-945ca8ebf299"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:14.970120 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.970094    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9" pod "0b77d4cb-5b07-448d-952a-945ca8ebf299" (UID: "0b77d4cb-5b07-448d-952a-945ca8ebf299"). InnerVolumeSpecName "kube-api-access-hwsm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:14.970218 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:14.970172    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca" pod "0b77d4cb-5b07-448d-952a-945ca8ebf299" (UID: "0b77d4cb-5b07-448d-952a-945ca8ebf299"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:15.069647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.069563    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.069647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.069609    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0b77d4cb-5b07-448d-952a-945ca8ebf299-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.069647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.069620    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwsm9\" (UniqueName: \"kubernetes.io/projected/0b77d4cb-5b07-448d-952a-945ca8ebf299-kube-api-access-hwsm9\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:15.112242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.112201    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"]
Oct 30 21:54:15.112942 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.112924    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0b77d4cb-5b07-448d-952a-945ca8ebf299" containerName="revoke-keys"
Oct 30 21:54:15.112994 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.112946    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0b77d4cb-5b07-448d-952a-945ca8ebf299" containerName="revoke-keys"
Oct 30 21:54:15.113070 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.113059    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0b77d4cb-5b07-448d-952a-945ca8ebf299" containerName="revoke-keys"
Oct 30 21:54:15.116910 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.116888    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.170480 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.170444    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.170480 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.170486    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvbwr\" (UniqueName: \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.170733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.170571    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"]
Oct 30 21:54:15.170733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.170615    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.271776 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.271737    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvbwr\" (UniqueName: \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.271996 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.271810    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.271996 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.271946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.272497 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.272475    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.274307 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.274282    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.333643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.333613    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvbwr\" (UniqueName: \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\") pod \"maas-api-revoke-sub-bbe53b4d-gpz9h\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.431630 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.431582    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:15.788051 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.787974    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"
Oct 30 21:54:15.788051 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.788010    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl" event={"ID":"0b77d4cb-5b07-448d-952a-945ca8ebf299","Type":"ContainerDied","Data":"bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115"}
Oct 30 21:54:15.788226 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.788078    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bc34125a8b3d8d9a44827c692d32807a24d4c91b94fab0c5eb115dc60e318115"
Oct 30 21:54:15.818404 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:15.818361    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod325b522f_e276_4f02_9cb2_ea34bc9869b1.slice/crio-afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64 WatchSource:0}: Error finding container afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64: Status 404 returned error can't find the container with id afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64
Oct 30 21:54:15.820816 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:15.820790    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"]
Oct 30 21:54:16.794174 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:16.794139    4882 generic.go:356] "Generic (PLEG): container finished" podID="325b522f-e276-4f02-9cb2-ea34bc9869b1" containerID="34c6fbc71f44d6a2892ea7771466202d2d646deb972592724c6d43ad0ccc9ec3" exitCode=0
Oct 30 21:54:16.794660 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:16.794223    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h" event={"ID":"325b522f-e276-4f02-9cb2-ea34bc9869b1","Type":"ContainerDied","Data":"34c6fbc71f44d6a2892ea7771466202d2d646deb972592724c6d43ad0ccc9ec3"}
Oct 30 21:54:16.794660 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:16.794281    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h" event={"ID":"325b522f-e276-4f02-9cb2-ea34bc9869b1","Type":"ContainerStarted","Data":"afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64"}
Oct 30 21:54:17.143651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.143613    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:54:17.946644 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.946612    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:17.997825 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.997778    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\") pod \"325b522f-e276-4f02-9cb2-ea34bc9869b1\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") "
Oct 30 21:54:17.997825 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.997835    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\" (UniqueName: \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\") pod \"325b522f-e276-4f02-9cb2-ea34bc9869b1\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") "
Oct 30 21:54:17.998108 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.997962    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\") pod \"325b522f-e276-4f02-9cb2-ea34bc9869b1\" (UID: \"325b522f-e276-4f02-9cb2-ea34bc9869b1\") "
Oct 30 21:54:17.998237 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.998203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca" pod "325b522f-e276-4f02-9cb2-ea34bc9869b1" (UID: "325b522f-e276-4f02-9cb2-ea34bc9869b1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:17.998320 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:17.998294    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:18.000092 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.000065    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr" pod "325b522f-e276-4f02-9cb2-ea34bc9869b1" (UID: "325b522f-e276-4f02-9cb2-ea34bc9869b1"). InnerVolumeSpecName "kube-api-access-pvbwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:18.000179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.000089    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca" pod "325b522f-e276-4f02-9cb2-ea34bc9869b1" (UID: "325b522f-e276-4f02-9cb2-ea34bc9869b1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:18.099086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.099048    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/325b522f-e276-4f02-9cb2-ea34bc9869b1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:18.099086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.099086    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvbwr\" (UniqueName: \"kubernetes.io/projected/325b522f-e276-4f02-9cb2-ea34bc9869b1-kube-api-access-pvbwr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:18.806399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.806358    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"
Oct 30 21:54:18.806399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.806389    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h" event={"ID":"325b522f-e276-4f02-9cb2-ea34bc9869b1","Type":"ContainerDied","Data":"afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64"}
Oct 30 21:54:18.806698 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:18.806440    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="afb7bfd2c66448bcb6753d68f0b69cc853c2194a48c002096edf4596e5ab6a64"
Oct 30 21:54:20.806718 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:54:20.806683    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:54:27.475564 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.475526    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"]
Oct 30 21:54:27.476032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.475963    4882 cpu_manager.go:414] "containerMap: removing container" podUID="325b522f-e276-4f02-9cb2-ea34bc9869b1" containerName="revoke-keys"
Oct 30 21:54:27.476032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.475975    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="325b522f-e276-4f02-9cb2-ea34bc9869b1" containerName="revoke-keys"
Oct 30 21:54:27.476119 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.476068    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="325b522f-e276-4f02-9cb2-ea34bc9869b1" containerName="revoke-keys"
Oct 30 21:54:27.479546 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.479528    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.533900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.533869    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:27.583718 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.583686    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:54:27.586011 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.585987    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"]
Oct 30 21:54:27.592025 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.591996    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.592152 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.592122    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.592196 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.592165    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pptlt\" (UniqueName: \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.693110 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.693068    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.693308 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.693172    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.693308 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.693222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pptlt\" (UniqueName: \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.693838 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.693819    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.695518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.695498    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.760636 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.760551    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pptlt\" (UniqueName: \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\") pod \"maas-api-revoke-sub-e5402be1-vbjkw\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:27.792438 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:27.792392    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:28.167684 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:28.167652    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddddbe74e_726c_4378_8854_b37007689716.slice/crio-29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7 WatchSource:0}: Error finding container 29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7: Status 404 returned error can't find the container with id 29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7
Oct 30 21:54:28.167831 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:28.167801    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"]
Oct 30 21:54:28.861018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:28.860980    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw" event={"ID":"dddbe74e-726c-4378-8854-b37007689716","Type":"ContainerStarted","Data":"46df8b94db7946ca905c0cb70967393e57351f59a029cd1ee7506bac0c592ee4"}
Oct 30 21:54:28.861435 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:28.861040    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw" event={"ID":"dddbe74e-726c-4378-8854-b37007689716","Type":"ContainerStarted","Data":"29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7"}
Oct 30 21:54:29.002919 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:29.002867    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw" podStartSLOduration=2.002850521 podStartE2EDuration="2.002850521s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:29.000074138 +0000 UTC m=+992.173398592" watchObservedRunningTime="2026-10-30 21:54:29.002850521 +0000 UTC m=+992.176174948"
Oct 30 21:54:29.867674 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:29.867635    4882 generic.go:356] "Generic (PLEG): container finished" podID="dddbe74e-726c-4378-8854-b37007689716" containerID="46df8b94db7946ca905c0cb70967393e57351f59a029cd1ee7506bac0c592ee4" exitCode=0
Oct 30 21:54:29.868135 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:29.867819    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw" event={"ID":"dddbe74e-726c-4378-8854-b37007689716","Type":"ContainerDied","Data":"46df8b94db7946ca905c0cb70967393e57351f59a029cd1ee7506bac0c592ee4"}
Oct 30 21:54:31.013027 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.013000    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:31.129790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.129704    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\") pod \"dddbe74e-726c-4378-8854-b37007689716\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") "
Oct 30 21:54:31.129790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.129769    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\" (UniqueName: \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\") pod \"dddbe74e-726c-4378-8854-b37007689716\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") "
Oct 30 21:54:31.129992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.129945    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\") pod \"dddbe74e-726c-4378-8854-b37007689716\" (UID: \"dddbe74e-726c-4378-8854-b37007689716\") "
Oct 30 21:54:31.130091 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.130067    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca" pod "dddbe74e-726c-4378-8854-b37007689716" (UID: "dddbe74e-726c-4378-8854-b37007689716"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:31.130287 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.130269    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dddbe74e-726c-4378-8854-b37007689716-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.131908 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.131883    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca" pod "dddbe74e-726c-4378-8854-b37007689716" (UID: "dddbe74e-726c-4378-8854-b37007689716"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:31.132013 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.131948    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt" pod "dddbe74e-726c-4378-8854-b37007689716" (UID: "dddbe74e-726c-4378-8854-b37007689716"). InnerVolumeSpecName "kube-api-access-pptlt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:31.231643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.231601    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dddbe74e-726c-4378-8854-b37007689716-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.231643 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.231635    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pptlt\" (UniqueName: \"kubernetes.io/projected/dddbe74e-726c-4378-8854-b37007689716-kube-api-access-pptlt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:31.878893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.878852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw" event={"ID":"dddbe74e-726c-4378-8854-b37007689716","Type":"ContainerDied","Data":"29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7"}
Oct 30 21:54:31.878893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.878880    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"
Oct 30 21:54:31.879124 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:31.878912    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29776ff867038bf4908f10d0cd84c15a119622f1997e6167faec932e990468f7"
Oct 30 21:54:35.907225 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.907188    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"]
Oct 30 21:54:35.907714 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.907697    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dddbe74e-726c-4378-8854-b37007689716" containerName="revoke-keys"
Oct 30 21:54:35.907769 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.907715    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dddbe74e-726c-4378-8854-b37007689716" containerName="revoke-keys"
Oct 30 21:54:35.907834 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.907825    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dddbe74e-726c-4378-8854-b37007689716" containerName="revoke-keys"
Oct 30 21:54:35.911145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.911128    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:35.969849 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.969807    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:35.970043 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:35.969947    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:54:36.023203 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.023168    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"]
Oct 30 21:54:36.081695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.081655    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.081929 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.081894    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vrbp6\" (UniqueName: \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.082075 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.082014    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.183427 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.183334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.183427 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.183387    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vrbp6\" (UniqueName: \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.183427 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.183408    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.184183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.184161    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.185783 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.185755    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.245880 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.245838    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vrbp6\" (UniqueName: \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\") pod \"maas-api-revoke-sub-a121c921-2c4lv\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.524341 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.524242    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:36.901480 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:36.901450    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod41db0538_37d9_4355_a3cd_ccef4b7d2f50.slice/crio-4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075 WatchSource:0}: Error finding container 4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075: Status 404 returned error can't find the container with id 4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075
Oct 30 21:54:36.904360 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:36.904328    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"]
Oct 30 21:54:37.911892 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:37.911857    4882 generic.go:356] "Generic (PLEG): container finished" podID="41db0538-37d9-4355-a3cd-ccef4b7d2f50" containerID="a990a0f94ab955e99a93951c432798264297e37890bf72e979c903ed9b7c6d00" exitCode=0
Oct 30 21:54:37.912282 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:37.911937    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv" event={"ID":"41db0538-37d9-4355-a3cd-ccef4b7d2f50","Type":"ContainerDied","Data":"a990a0f94ab955e99a93951c432798264297e37890bf72e979c903ed9b7c6d00"}
Oct 30 21:54:37.912282 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:37.911990    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv" event={"ID":"41db0538-37d9-4355-a3cd-ccef4b7d2f50","Type":"ContainerStarted","Data":"4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075"}
Oct 30 21:54:39.057314 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.057284    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:39.116890 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.116856    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\") pod \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") "
Oct 30 21:54:39.117057 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.116931    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\" (UniqueName: \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\") pod \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") "
Oct 30 21:54:39.117057 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.116977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\") pod \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\" (UID: \"41db0538-37d9-4355-a3cd-ccef4b7d2f50\") "
Oct 30 21:54:39.117385 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.117360    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca" pod "41db0538-37d9-4355-a3cd-ccef4b7d2f50" (UID: "41db0538-37d9-4355-a3cd-ccef4b7d2f50"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:39.119030 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.119004    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca" pod "41db0538-37d9-4355-a3cd-ccef4b7d2f50" (UID: "41db0538-37d9-4355-a3cd-ccef4b7d2f50"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:39.119147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.119131    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6" pod "41db0538-37d9-4355-a3cd-ccef4b7d2f50" (UID: "41db0538-37d9-4355-a3cd-ccef4b7d2f50"). InnerVolumeSpecName "kube-api-access-vrbp6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:39.217900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.217806    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.217900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.217843    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vrbp6\" (UniqueName: \"kubernetes.io/projected/41db0538-37d9-4355-a3cd-ccef4b7d2f50-kube-api-access-vrbp6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.217900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.217868    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/41db0538-37d9-4355-a3cd-ccef4b7d2f50-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:39.925416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.925383    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"
Oct 30 21:54:39.925626 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.925384    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv" event={"ID":"41db0538-37d9-4355-a3cd-ccef4b7d2f50","Type":"ContainerDied","Data":"4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075"}
Oct 30 21:54:39.925626 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:39.925505    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4b2a3d0299c22f4b10fa469721a3d6b4d4e1976f5f86210316a9c2120f85b075"
Oct 30 21:54:46.599436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.599401    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"]
Oct 30 21:54:46.599964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.599922    4882 cpu_manager.go:414] "containerMap: removing container" podUID="41db0538-37d9-4355-a3cd-ccef4b7d2f50" containerName="revoke-keys"
Oct 30 21:54:46.599964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.599935    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="41db0538-37d9-4355-a3cd-ccef4b7d2f50" containerName="revoke-keys"
Oct 30 21:54:46.600046 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.600009    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="41db0538-37d9-4355-a3cd-ccef4b7d2f50" containerName="revoke-keys"
Oct 30 21:54:46.603182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.603166    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.656239 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.656208    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:46.691641 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.691573    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfbb7\" (UniqueName: \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.691828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.691739    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.691828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.691801    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.706553 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.706525    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:54:46.708974 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.708954    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"]
Oct 30 21:54:46.792417 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.792384    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfbb7\" (UniqueName: \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.792633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.792448    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.792633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.792487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.793115 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.793092    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.794859 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.794842    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.854563 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.854488    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfbb7\" (UniqueName: \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\") pod \"maas-api-revoke-sub-a0e95e13-zhr7v\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:46.916172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:46.916143    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:47.292544 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:47.292507    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52fd999b_d1d5_4248_914e_f4cf525b7886.slice/crio-232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d WatchSource:0}: Error finding container 232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d: Status 404 returned error can't find the container with id 232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d
Oct 30 21:54:47.294948 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.294921    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"]
Oct 30 21:54:47.369059 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.369024    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:54:47.374306 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.374282    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.427701 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.427656    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-plugins\""
Oct 30 21:54:47.428337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.428308    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:54:47.475420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.475377    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-fd60520bcb62\""
Oct 30 21:54:47.475420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.475377    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"agc-cred-7653478ffe43\""
Oct 30 21:54:47.475420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.475415    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"routing-overlay\""
Oct 30 21:54:47.475420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.475404    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"kube-root-ca.crt\""
Oct 30 21:54:47.477338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.477322    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"models-as-a-service\"/\"payload-processing-external-model-dockercfg-pjjkg\""
Oct 30 21:54:47.477428 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.477386    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"models-as-a-service\"/\"openshift-service-ca.crt\""
Oct 30 21:54:47.501029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.500999    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f92n6\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.501173 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.501045    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.501215 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.501179    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.501250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.501229    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602012 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.601979    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602527 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.602033    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602527 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.602074    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f92n6\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602527 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.602110    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.602861    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.602997 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.602976    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.604620 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.604580    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.666136 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.666105    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f92n6\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\") pod \"payload-processing-external-model-f854549f8-rptzn\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") " pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.687056 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.687028    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:47.969616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.969554    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v" event={"ID":"52fd999b-d1d5-4248-914e-f4cf525b7886","Type":"ContainerStarted","Data":"5f0c2a6c114d5d4efdd68d01044cdce213d0a637a02425fbacb930816eb2c878"}
Oct 30 21:54:47.969850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:47.969638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v" event={"ID":"52fd999b-d1d5-4248-914e-f4cf525b7886","Type":"ContainerStarted","Data":"232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d"}
Oct 30 21:54:48.064260 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:54:48.064185    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod197457cc_b2ea_47fb_8acf_b8f4a84c4ce2.slice/crio-f7499078fa9ea2ddd92100bff6a6336e9d35efa2bc139e920fcd4dc0cc97117a WatchSource:0}: Error finding container f7499078fa9ea2ddd92100bff6a6336e9d35efa2bc139e920fcd4dc0cc97117a: Status 404 returned error can't find the container with id f7499078fa9ea2ddd92100bff6a6336e9d35efa2bc139e920fcd4dc0cc97117a
Oct 30 21:54:48.067709 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.067681    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:54:48.979178 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.979146    4882 generic.go:356] "Generic (PLEG): container finished" podID="52fd999b-d1d5-4248-914e-f4cf525b7886" containerID="5f0c2a6c114d5d4efdd68d01044cdce213d0a637a02425fbacb930816eb2c878" exitCode=0
Oct 30 21:54:48.979691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.979249    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v" event={"ID":"52fd999b-d1d5-4248-914e-f4cf525b7886","Type":"ContainerDied","Data":"5f0c2a6c114d5d4efdd68d01044cdce213d0a637a02425fbacb930816eb2c878"}
Oct 30 21:54:48.981541 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.981506    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" event={"ID":"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2","Type":"ContainerStarted","Data":"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"}
Oct 30 21:54:48.981691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.981570    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" event={"ID":"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2","Type":"ContainerStarted","Data":"f7499078fa9ea2ddd92100bff6a6336e9d35efa2bc139e920fcd4dc0cc97117a"}
Oct 30 21:54:48.981691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:48.981618    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:54:49.216039 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:49.215991    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" podStartSLOduration=2.215972514 podStartE2EDuration="2.215972514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:49.215632843 +0000 UTC m=+1012.388957271" watchObservedRunningTime="2026-10-30 21:54:49.215972514 +0000 UTC m=+1012.389296934"
Oct 30 21:54:50.148629 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.148599    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:50.228772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.228741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\") pod \"52fd999b-d1d5-4248-914e-f4cf525b7886\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") "
Oct 30 21:54:50.228916 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.228838    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\") pod \"52fd999b-d1d5-4248-914e-f4cf525b7886\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") "
Oct 30 21:54:50.228916 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.228888    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\" (UniqueName: \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\") pod \"52fd999b-d1d5-4248-914e-f4cf525b7886\" (UID: \"52fd999b-d1d5-4248-914e-f4cf525b7886\") "
Oct 30 21:54:50.229066 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.229046    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca" pod "52fd999b-d1d5-4248-914e-f4cf525b7886" (UID: "52fd999b-d1d5-4248-914e-f4cf525b7886"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:50.229264 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.229249    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.231065 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.231043    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca" pod "52fd999b-d1d5-4248-914e-f4cf525b7886" (UID: "52fd999b-d1d5-4248-914e-f4cf525b7886"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:50.231125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.231051    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7" pod "52fd999b-d1d5-4248-914e-f4cf525b7886" (UID: "52fd999b-d1d5-4248-914e-f4cf525b7886"). InnerVolumeSpecName "kube-api-access-pfbb7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:50.330568 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.330542    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52fd999b-d1d5-4248-914e-f4cf525b7886-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.331030 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.330571    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfbb7\" (UniqueName: \"kubernetes.io/projected/52fd999b-d1d5-4248-914e-f4cf525b7886-kube-api-access-pfbb7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:50.808837 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:54:50.808749    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:54:50.995419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.995393    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"
Oct 30 21:54:50.995608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.995430    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v" event={"ID":"52fd999b-d1d5-4248-914e-f4cf525b7886","Type":"ContainerDied","Data":"232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d"}
Oct 30 21:54:50.995608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:50.995476    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="232e72e43b79baf9677c7fa18e0b82a6eb1c990ef8839e626881746145cddb7d"
Oct 30 21:54:52.726926 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.726883    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:54:52.757656 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.757616    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"]
Oct 30 21:54:52.758161 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.758146    4882 cpu_manager.go:414] "containerMap: removing container" podUID="52fd999b-d1d5-4248-914e-f4cf525b7886" containerName="revoke-keys"
Oct 30 21:54:52.758209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.758163    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52fd999b-d1d5-4248-914e-f4cf525b7886" containerName="revoke-keys"
Oct 30 21:54:52.758255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.758246    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="52fd999b-d1d5-4248-914e-f4cf525b7886" containerName="revoke-keys"
Oct 30 21:54:52.761827 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.761810    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.821495 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.821460    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:54:52.821688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.821498    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:54:52.835538 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.835506    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"]
Oct 30 21:54:52.856997 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.856960    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvtlv\" (UniqueName: \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.857148 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.857009    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.857260 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.857237    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.958083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.958045    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.958255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.958115    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvtlv\" (UniqueName: \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.958255 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.958156    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.958773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.958748    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:52.960568 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:52.960550    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:53.020986 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:53.020913    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvtlv\" (UniqueName: \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\") pod \"maas-api-revoke-sub-03173270-v7gr2\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:53.075350 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:53.075316    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:53.452928 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:53.452897    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"]
Oct 30 21:54:54.013757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:54.013658    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2" event={"ID":"7241e241-b3d4-4efd-9848-5a9841dfc48e","Type":"ContainerStarted","Data":"9bb6897164fe8f9c259729c34bae9e409704bd448caee3723a24dad79da52159"}
Oct 30 21:54:54.013757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:54.013715    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2" event={"ID":"7241e241-b3d4-4efd-9848-5a9841dfc48e","Type":"ContainerStarted","Data":"a34d134830298c785d5499a0f6ae400a809cdc70b4303d01627476d4b3068f43"}
Oct 30 21:54:54.132905 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:54.132835    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2" podStartSLOduration=2.132812259 podStartE2EDuration="2.132812259s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:54:54.130556584 +0000 UTC m=+1017.303881011" watchObservedRunningTime="2026-10-30 21:54:54.132812259 +0000 UTC m=+1017.306136688"
Oct 30 21:54:55.020695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:55.020661    4882 generic.go:356] "Generic (PLEG): container finished" podID="7241e241-b3d4-4efd-9848-5a9841dfc48e" containerID="9bb6897164fe8f9c259729c34bae9e409704bd448caee3723a24dad79da52159" exitCode=0
Oct 30 21:54:55.021097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:55.020749    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2" event={"ID":"7241e241-b3d4-4efd-9848-5a9841dfc48e","Type":"ContainerDied","Data":"9bb6897164fe8f9c259729c34bae9e409704bd448caee3723a24dad79da52159"}
Oct 30 21:54:56.164538 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.164513    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:56.295461 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.295375    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\") pod \"7241e241-b3d4-4efd-9848-5a9841dfc48e\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") "
Oct 30 21:54:56.295627 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.295496    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\") pod \"7241e241-b3d4-4efd-9848-5a9841dfc48e\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") "
Oct 30 21:54:56.295627 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.295538    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\" (UniqueName: \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\") pod \"7241e241-b3d4-4efd-9848-5a9841dfc48e\" (UID: \"7241e241-b3d4-4efd-9848-5a9841dfc48e\") "
Oct 30 21:54:56.295768 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.295744    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca" pod "7241e241-b3d4-4efd-9848-5a9841dfc48e" (UID: "7241e241-b3d4-4efd-9848-5a9841dfc48e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:54:56.295965 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.295940    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.297557 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.297528    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca" pod "7241e241-b3d4-4efd-9848-5a9841dfc48e" (UID: "7241e241-b3d4-4efd-9848-5a9841dfc48e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:54:56.297699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.297613    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv" pod "7241e241-b3d4-4efd-9848-5a9841dfc48e" (UID: "7241e241-b3d4-4efd-9848-5a9841dfc48e"). InnerVolumeSpecName "kube-api-access-vvtlv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:54:56.397130 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.397085    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7241e241-b3d4-4efd-9848-5a9841dfc48e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:56.397130 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:56.397121    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvtlv\" (UniqueName: \"kubernetes.io/projected/7241e241-b3d4-4efd-9848-5a9841dfc48e-kube-api-access-vvtlv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:54:57.032163 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:57.032130    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"
Oct 30 21:54:57.032329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:57.032158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2" event={"ID":"7241e241-b3d4-4efd-9848-5a9841dfc48e","Type":"ContainerDied","Data":"a34d134830298c785d5499a0f6ae400a809cdc70b4303d01627476d4b3068f43"}
Oct 30 21:54:57.032329 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:57.032230    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34d134830298c785d5499a0f6ae400a809cdc70b4303d01627476d4b3068f43"
Oct 30 21:54:59.992144 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:54:59.992115    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:55:12.315711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.315670    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:55:12.316182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.316168    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7241e241-b3d4-4efd-9848-5a9841dfc48e" containerName="revoke-keys"
Oct 30 21:55:12.316239 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.316182    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7241e241-b3d4-4efd-9848-5a9841dfc48e" containerName="revoke-keys"
Oct 30 21:55:12.316278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.316261    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7241e241-b3d4-4efd-9848-5a9841dfc48e" containerName="revoke-keys"
Oct 30 21:55:12.320736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.320714    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.378276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.378238    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:55:12.424996 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.424961    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"e2e-conflict-a-09506444-kserve-self-signed-certs\""
Oct 30 21:55:12.467040 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467003    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.467210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467144    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.467210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.467337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467275    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.467404 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467379    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hrpg\" (UniqueName: \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.467450 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.467439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569011    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569113 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569090    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hrpg\" (UniqueName: \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569139    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569355 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569329    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569532 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569355    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.569853 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569825    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.570021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.569996    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.570198 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.570175    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.572411 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.572362    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.572539 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.572472    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.631752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.631712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hrpg\" (UniqueName: \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\") pod \"e2e-conflict-a-09506444-kserve-5f65586799-bv8r9\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") " pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:12.636536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:12.636270    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:13.037050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:13.037019    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:55:13.037636 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:55:13.037609    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode6a8b679_1a6e_4ccc_ab8b_576055e6843b.slice/crio-14aa6a89ea9bd41c6c0047a6143523ec12c544503a6185d81d5a54fc7e9ae423 WatchSource:0}: Error finding container 14aa6a89ea9bd41c6c0047a6143523ec12c544503a6185d81d5a54fc7e9ae423: Status 404 returned error can't find the container with id 14aa6a89ea9bd41c6c0047a6143523ec12c544503a6185d81d5a54fc7e9ae423
Oct 30 21:55:13.120237 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:13.120199    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" event={"ID":"e6a8b679-1a6e-4ccc-ab8b-576055e6843b","Type":"ContainerStarted","Data":"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"}
Oct 30 21:55:13.120237 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:13.120239    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" event={"ID":"e6a8b679-1a6e-4ccc-ab8b-576055e6843b","Type":"ContainerStarted","Data":"14aa6a89ea9bd41c6c0047a6143523ec12c544503a6185d81d5a54fc7e9ae423"}
Oct 30 21:55:13.636373 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:13.636339    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:15.059696 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.059647    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" podStartSLOduration=3.059632093 podStartE2EDuration="3.059632093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:13.240745279 +0000 UTC m=+1036.414069705" watchObservedRunningTime="2026-10-30 21:55:15.059632093 +0000 UTC m=+1038.232956520"
Oct 30 21:55:15.060072 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.060031    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 21:55:15.063818 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.063796    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.176847 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.176811    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 21:55:15.194434 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194392    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194434 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194648    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-488wz\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194692    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194965 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194965 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194802    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.194965 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.194833    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295435 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295395    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295658 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295547    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295706 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295658    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295706 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295678    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-488wz\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295798 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295730    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295811    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295828    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.295900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.296047 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295914    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.296047 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.295972    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.296285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.296266    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.296369 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.296349    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.296422 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.296403    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.298210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.298181    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.298445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.298428    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.361321 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.361281    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.361898 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.361876    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-488wz\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\") pod \"maas-default-gateway-openshift-default-7df854bf67-s4ndm\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.377823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.377793    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:15.804291 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:55:15.804254    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode617471f_8fa4_4e4e_ad62_f4e9b613bd52.slice/crio-a2bb89e70624292504578ac37484155f839e805e6d09535819ac33c0a8e8cab4 WatchSource:0}: Error finding container a2bb89e70624292504578ac37484155f839e805e6d09535819ac33c0a8e8cab4: Status 404 returned error can't find the container with id a2bb89e70624292504578ac37484155f839e805e6d09535819ac33c0a8e8cab4
Oct 30 21:55:15.806713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.806687    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 21:55:15.807642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.807569    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:15.807733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.807673    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:15.807733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:15.807707    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:16.139559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.139521    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" event={"ID":"e617471f-8fa4-4e4e-ad62-f4e9b613bd52","Type":"ContainerStarted","Data":"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"}
Oct 30 21:55:16.140014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.139578    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" event={"ID":"e617471f-8fa4-4e4e-ad62-f4e9b613bd52","Type":"ContainerStarted","Data":"a2bb89e70624292504578ac37484155f839e805e6d09535819ac33c0a8e8cab4"}
Oct 30 21:55:16.265315 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.265244    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podStartSLOduration=1.265224034 podStartE2EDuration="1.265224034s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:16.264112426 +0000 UTC m=+1039.437436854" watchObservedRunningTime="2026-10-30 21:55:16.265224034 +0000 UTC m=+1039.438548462"
Oct 30 21:55:16.378707 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.378671    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:16.380412 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.380383    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:16.380543 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:16.380437    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:17.378847 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:17.378811    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:17.379247 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:17.378862    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:18.378833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:18.378794    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:18.379106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:18.378851    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:19.378440 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:19.378398    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:19.378654 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:19.378462    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:20.378999 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:20.378966    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:20.379449 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:20.379018    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:20.807901 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:55:20.807823    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:55:21.378866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:21.378825    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:21.379107 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:21.378891    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:22.379176 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:22.379138    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:22.379606 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:22.379205    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:22.637500 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:22.637407    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:22.649777 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:22.649741    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:23.196425 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:23.196397    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:55:23.378819 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:23.378775    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Oct 30 21:55:23.379037 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:23.378843    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Oct 30 21:55:24.606969 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:24.606927    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:24.607567 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:24.607541    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:24.609007 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:24.608974    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 21:55:26.068153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:26.068118    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:55:30.040808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.040776    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 21:55:30.046083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.046059    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170278 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170233    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170486 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170290    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170486 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170331    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170486 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170380    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170486 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170432    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd7gv\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170505    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170532    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170557    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.170756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.170695    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.243859 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.243818    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 21:55:30.272328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272293    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272332    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272598 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272349    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272598 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272476    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272602    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272634    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272680    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272742    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272805    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd7gv\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272973 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272890    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.272973 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272945    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.273075 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.272983    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.273075 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.273021    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.273461 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.273442    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.274826 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.274809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.275254 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.275232    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.335902 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.335852    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.336060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.335895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd7gv\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\") pod \"maas-default-gateway-openshift-default-7df854bf67-4kb4v\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.361428 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.361396    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:30.786434 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:55:30.786374    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c11bb7e_267e_405e_94ba_9b28e6a7bd68.slice/crio-20c11aef974ab235c02b12febc5e9a819f30a25fa78bad772ade55c2af0f3c9a WatchSource:0}: Error finding container 20c11aef974ab235c02b12febc5e9a819f30a25fa78bad772ade55c2af0f3c9a: Status 404 returned error can't find the container with id 20c11aef974ab235c02b12febc5e9a819f30a25fa78bad772ade55c2af0f3c9a
Oct 30 21:55:30.789133 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.789096    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:30.789249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.789182    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:30.789249 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.789229    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:30.789363 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:30.789287    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 21:55:31.235114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.235075    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" event={"ID":"0c11bb7e-267e-405e-94ba-9b28e6a7bd68","Type":"ContainerStarted","Data":"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"}
Oct 30 21:55:31.235644 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.235176    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" event={"ID":"0c11bb7e-267e-405e-94ba-9b28e6a7bd68","Type":"ContainerStarted","Data":"20c11aef974ab235c02b12febc5e9a819f30a25fa78bad772ade55c2af0f3c9a"}
Oct 30 21:55:31.362165 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.362136    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:31.363621 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.363576    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:31.363675 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.363654    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:31.464378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:31.464320    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podStartSLOduration=2.464306389 podStartE2EDuration="2.464306389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:31.412525689 +0000 UTC m=+1054.585850116" watchObservedRunningTime="2026-10-30 21:55:31.464306389 +0000 UTC m=+1054.637630815"
Oct 30 21:55:32.362212 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:32.362176    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:32.362734 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:32.362236    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:33.362957 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:33.362917    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:33.363365 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:33.362969    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:34.362880 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:34.362730    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:34.362880 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:34.362784    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:35.361728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:35.361693    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:35.361895 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:35.361760    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:36.362208 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:36.362176    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:36.362626 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:36.362228    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:37.362690 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:37.362646    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:37.363077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:37.362713    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:38.362179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:38.362146    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Oct 30 21:55:38.362370 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:38.362199    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Oct 30 21:55:39.641987 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:39.641949    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:39.642518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:39.642365    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:39.643429 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:39.643407    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 21:55:50.806488 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:55:50.806445    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:55:55.607018 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.606985    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 21:55:55.611492 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.611469    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.668028 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.667992    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3c4802-openshift-default-dockercfg-6gcqv\""
Oct 30 21:55:55.719563 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.719528    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 21:55:55.771451 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771410    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771463    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771560    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771642    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771683    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjl45\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771808 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771756    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771888 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771812    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771888 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771834    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.771959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.771947    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872548    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872722    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872782    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872819    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872855    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjl45\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.872992 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.872964    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873154 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873000    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873154 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873082    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873263 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873193    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873323 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873281    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873559    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873643    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.873711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.873668    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.875172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.875143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.875268 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.875207    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.934964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.934935    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:55.935498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:55.935473    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjl45\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:56.225557 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:56.225478    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:56.658630 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:55:56.658581    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10c61a5f_f9f1_448f_a5f1_f371e0444500.slice/crio-4169f17d6cbe80bf00bfb565c980e20d731d2dc2fbc16a923c947ab8649808a0 WatchSource:0}: Error finding container 4169f17d6cbe80bf00bfb565c980e20d731d2dc2fbc16a923c947ab8649808a0: Status 404 returned error can't find the container with id 4169f17d6cbe80bf00bfb565c980e20d731d2dc2fbc16a923c947ab8649808a0
Oct 30 21:55:56.660375 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:56.660338    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:56.660457 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:56.660416    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:56.660457 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:56.660447    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:55:56.660770 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:56.660746    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 21:55:57.403610 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:57.403563    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" event={"ID":"10c61a5f-f9f1-448f-a5f1-f371e0444500","Type":"ContainerStarted","Data":"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"}
Oct 30 21:55:57.403826 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:57.403627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" event={"ID":"10c61a5f-f9f1-448f-a5f1-f371e0444500","Type":"ContainerStarted","Data":"4169f17d6cbe80bf00bfb565c980e20d731d2dc2fbc16a923c947ab8649808a0"}
Oct 30 21:55:57.629544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:57.629460    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podStartSLOduration=2.629441028 podStartE2EDuration="2.629441028s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:55:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:55:57.576085037 +0000 UTC m=+1080.749409464" watchObservedRunningTime="2026-10-30 21:55:57.629441028 +0000 UTC m=+1080.802765456"
Oct 30 21:55:58.231266 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:58.229940    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:55:58.235616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:58.233100    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:55:58.235616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:58.233167    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:55:59.236612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:59.231045    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:55:59.236612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:55:59.231108    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:56:00.237004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:00.236965    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:56:00.237664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:00.237627    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:56:01.251111 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:01.251068    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:56:01.251536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:01.251140    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:56:02.244618 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:02.243178    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:56:02.244618 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:02.243241    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:56:03.226806 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:03.226770    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Oct 30 21:56:03.227313 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:03.226833    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Oct 30 21:56:04.230742 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:04.230713    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:56:04.231135 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:04.230988    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:56:04.231711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:04.231692    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 21:56:04.983523 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:04.983486    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 21:56:04.987686 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:04.987667    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.041455 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.041399    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"openshift-service-ca.crt\""
Oct 30 21:56:05.041780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.041541    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"default-dockercfg-7bwxd\""
Oct 30 21:56:05.041780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.041735    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"distinct-w3-4d14e0-kserve-self-signed-certs\""
Oct 30 21:56:05.041780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.041777    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"kube-root-ca.crt\""
Oct 30 21:56:05.044480 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.044451    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 21:56:05.102186 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102153    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.102186 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102197    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.102414 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102293    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbpgg\" (UniqueName: \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.102414 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102334    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.102491 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102429    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.102533 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.102503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.203805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.203769    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.203832    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.203878    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.203929    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbpgg\" (UniqueName: \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.203962    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.204015    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204429 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.204394    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.204475    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.204542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.204515    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.206258 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.206226    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.206618 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.206578    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.269199 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.269106    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbpgg\" (UniqueName: \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\") pod \"distinct-w3-4d14e0-kserve-768fcfdc4b-zczff\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.299897 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.299830    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:05.695310 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:56:05.695277    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc83340a0_e2a0_4abc_87fd_58889e90a911.slice/crio-f168316b2e8e71900e277759373f5809af4db4434ffaf7a6443f675a66fc6717 WatchSource:0}: Error finding container f168316b2e8e71900e277759373f5809af4db4434ffaf7a6443f675a66fc6717: Status 404 returned error can't find the container with id f168316b2e8e71900e277759373f5809af4db4434ffaf7a6443f675a66fc6717
Oct 30 21:56:05.699542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:05.699491    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 21:56:06.384750 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:06.384712    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:56:06.479470 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:06.479416    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" event={"ID":"c83340a0-e2a0-4abc-87fd-58889e90a911","Type":"ContainerStarted","Data":"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"}
Oct 30 21:56:06.479672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:06.479484    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" event={"ID":"c83340a0-e2a0-4abc-87fd-58889e90a911","Type":"ContainerStarted","Data":"f168316b2e8e71900e277759373f5809af4db4434ffaf7a6443f675a66fc6717"}
Oct 30 21:56:06.605012 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:06.604958    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podStartSLOduration=2.604939336 podStartE2EDuration="2.604939336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:56:06.602079412 +0000 UTC m=+1089.775403838" watchObservedRunningTime="2026-10-30 21:56:06.604939336 +0000 UTC m=+1089.778263820"
Oct 30 21:56:07.300299 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:07.300264    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:15.300817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:15.300772    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:15.314739 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:15.314707    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:15.541192 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:15.541159    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:56:20.808017 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:20.807976    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:56:24.012453 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.012415    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7653478ffe43: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:24.012453 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.012455    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:24.013018 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.012542    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:24.512519695 +0000 UTC m=+1107.685844122 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:24.517942 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.517906    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7653478ffe43: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:24.517942 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.517936    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:24.518166 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:24.517991    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:25.517977023 +0000 UTC m=+1108.691301429 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:25.529723 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:25.529687    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7653478ffe43: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:25.529723 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:25.529721    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:25.530243 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:25.529776    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:27.529760789 +0000 UTC m=+1110.703085194 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:27.553720 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:27.553683    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-7653478ffe43: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:27.553720 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:27.553714    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:27.554176 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:27.553768    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:31.553754505 +0000 UTC m=+1114.727078910 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-7653478ffe43" not found
Oct 30 21:56:27.900759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:27.900722    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:56:31.599926 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:31.599895    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fd60520bcb62: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:31.599926 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:31.599921    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:31.600381 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:31.599972    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:39.599956414 +0000 UTC m=+1122.773280819 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:39.600265 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:39.600227    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fd60520bcb62: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:39.600265 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:39.600258    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:39.600738 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:39.600343    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:56:55.60032531 +0000 UTC m=+1138.773649714 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:50.807944 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:50.807897    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:56:51.345085 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:56:51.345041    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:56:55.621926 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:55.621892    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fd60520bcb62: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:55.621926 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:55.621920    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:56:55.622460 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:56:55.621999    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:57:27.621979207 +0000 UTC m=+1170.795303622 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-fd60520bcb62" not found
Oct 30 21:57:00.895054 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.895017    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 21:57:00.899335 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.899313    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.955020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.954987    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 21:57:00.980127 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980092    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfd79\" (UniqueName: \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.980339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980227    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.980339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.980471 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980376    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.980537 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980486    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:00.980537 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:00.980521    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.001157 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.001117    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-56f3fe\"/\"distinct-w2-56f3fe-kserve-self-signed-certs\""
Oct 30 21:57:01.081986 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.081949    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082059    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082179 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082118    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfd79\" (UniqueName: \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082353 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082210    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082353 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082545 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082574    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.082635 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.082618    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.084875 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.084855    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.085121 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.085103    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.144150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.144115    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfd79\" (UniqueName: \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\") pod \"distinct-w2-56f3fe-kserve-8575687f54-zrhpr\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") " pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.212406 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.212328    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:01.589430 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:57:01.589393    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod186a9dc2_ff2f_47d4_bb0a_6f4678b7dc16.slice/crio-9ee2158a9e09213d51217bd12207818a2208b62247a8bfefbb63b78d7d170878 WatchSource:0}: Error finding container 9ee2158a9e09213d51217bd12207818a2208b62247a8bfefbb63b78d7d170878: Status 404 returned error can't find the container with id 9ee2158a9e09213d51217bd12207818a2208b62247a8bfefbb63b78d7d170878
Oct 30 21:57:01.590998 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.590972    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 21:57:01.818522 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.818477    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" event={"ID":"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16","Type":"ContainerStarted","Data":"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"}
Oct 30 21:57:01.818522 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.818524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" event={"ID":"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16","Type":"ContainerStarted","Data":"9ee2158a9e09213d51217bd12207818a2208b62247a8bfefbb63b78d7d170878"}
Oct 30 21:57:01.938975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:01.938872    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podStartSLOduration=1.938853975 podStartE2EDuration="1.938853975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:01.936325343 +0000 UTC m=+1145.109649769" watchObservedRunningTime="2026-10-30 21:57:01.938853975 +0000 UTC m=+1145.112178447"
Oct 30 21:57:02.213131 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:02.213049    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:11.212699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:11.212664    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:11.226453 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:11.226423    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:11.889991 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:11.889959    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:57:13.681310 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:13.681274    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:57:20.808537 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:57:20.808487    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:57:26.956359 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:26.956322    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 21:57:26.962079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:26.962058    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018045    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018090    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018224    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vthgr\" (UniqueName: \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018265    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018288    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.018506 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.018484    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.065757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.065722    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-4d14e0\"/\"distinct-2-w3-4d14e0-kserve-self-signed-certs\""
Oct 30 21:57:27.069039 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.068995    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 21:57:27.120049 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120002    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vthgr\" (UniqueName: \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120055    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120088    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120342 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120322    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120390 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120380    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120452 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120402    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120580 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120557    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120749 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120723    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.120873 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.120781    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.122797 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.122770    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.123083 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.123067    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.182509 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.182476    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vthgr\" (UniqueName: \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\") pod \"distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") " pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.275625 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.275525    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:27.627480 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:57:27.627448    4882 projected.go:266] Couldn't get secret models-as-a-service/agc-cred-fd60520bcb62: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:57:27.627480 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:57:27.627475    4882 projected.go:196] Error preparing data for projected volume provider-credentials for pod models-as-a-service/payload-processing-external-model-f854549f8-rptzn: secret "agc-cred-fd60520bcb62" not found
Oct 30 21:57:27.627725 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:57:27.627538    4882 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials podName:197457cc-b2ea-47fb-8acf-b8f4a84c4ce2 nodeName:}" failed. No retries permitted until 2026-10-30 21:58:31.627522228 +0000 UTC m=+1234.800846634 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "provider-credentials" (UniqueName: "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials") pod "payload-processing-external-model-f854549f8-rptzn" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2") : secret "agc-cred-fd60520bcb62" not found
Oct 30 21:57:27.664937 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:57:27.664899    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb8c052ee_942c_4e66_a385_12e5f90f16e3.slice/crio-115b8646179bb5d76464af99b0f0f2040fb3fabe087e0500b9129c2c801e9d26 WatchSource:0}: Error finding container 115b8646179bb5d76464af99b0f0f2040fb3fabe087e0500b9129c2c801e9d26: Status 404 returned error can't find the container with id 115b8646179bb5d76464af99b0f0f2040fb3fabe087e0500b9129c2c801e9d26
Oct 30 21:57:27.667256 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.667232    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 21:57:27.989573 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.989470    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" event={"ID":"b8c052ee-942c-4e66-a385-12e5f90f16e3","Type":"ContainerStarted","Data":"8e1e41e2e64ddfa0a43b18d38aee9bde41f4e9b7861df910190c41378528c2f8"}
Oct 30 21:57:27.989573 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:27.989517    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" event={"ID":"b8c052ee-942c-4e66-a385-12e5f90f16e3","Type":"ContainerStarted","Data":"115b8646179bb5d76464af99b0f0f2040fb3fabe087e0500b9129c2c801e9d26"}
Oct 30 21:57:28.109465 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:28.109410    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podStartSLOduration=2.109394778 podStartE2EDuration="2.109394778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:57:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:57:28.107327853 +0000 UTC m=+1171.280652279" watchObservedRunningTime="2026-10-30 21:57:28.109394778 +0000 UTC m=+1171.282719205"
Oct 30 21:57:28.275712 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:28.275634    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:37.276490 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:37.276399    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:37.288873 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:37.288846    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:37.721323 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:37.721283    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:57:38.058925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:38.058847    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:57:50.809523 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:57:50.809486    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:57:56.241104 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.241072    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"]
Oct 30 21:57:56.245134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.245113    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.299729 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.299694    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"]
Oct 30 21:57:56.345966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.345929    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:57:56.345966 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.345947    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:57:56.362501 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.362472    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skxjq\" (UniqueName: \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.362699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.362558    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.362773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.362759    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.463621 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.463556    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.463835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.463689    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skxjq\" (UniqueName: \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.463835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.463772    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.464418 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.464394    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.465901 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.465883    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.522916 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.522830    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"]
Oct 30 21:57:56.526868 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.526850    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.534243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.534213    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skxjq\" (UniqueName: \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\") pod \"maas-api-revoke-sub-341eb523-pjjtv\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.559106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.559083    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:56.584216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.584180    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"]
Oct 30 21:57:56.665954 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.665917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.666127 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.666037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.666198 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.666184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jwjn\" (UniqueName: \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.767695 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.767649    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jwjn\" (UniqueName: \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.767904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.767717    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.767904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.767806    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.768494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.768469    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.770173 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.770151    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.829522 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.829458    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jwjn\" (UniqueName: \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\") pod \"maas-api-revoke-sub-9e25d20d-7r5nb\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.852350 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.852319    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:56.939953 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:57:56.939257    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0e060a_39a4_4b3e_939d_555a1d40122e.slice/crio-9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f WatchSource:0}: Error finding container 9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f: Status 404 returned error can't find the container with id 9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f
Oct 30 21:57:56.944230 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:56.944208    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"]
Oct 30 21:57:57.071368 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:57.071335    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:57:57.076734 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:57.076709    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 21:57:57.174762 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:57.174722    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv" event={"ID":"da0e060a-39a4-4b3e-939d-555a1d40122e","Type":"ContainerStarted","Data":"9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f"}
Oct 30 21:57:57.226379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:57.226343    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"]
Oct 30 21:57:57.227191 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:57:57.227166    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod818304b9_ec45_4dab_99cb_3b9b9df2e62b.slice/crio-0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64 WatchSource:0}: Error finding container 0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64: Status 404 returned error can't find the container with id 0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64
Oct 30 21:57:58.184773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:58.184671    4882 generic.go:356] "Generic (PLEG): container finished" podID="da0e060a-39a4-4b3e-939d-555a1d40122e" containerID="8e0af14b0b29921be3ec444c8aa19077f2a16ada380d95ac6500b66309589b42" exitCode=0
Oct 30 21:57:58.185184 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:58.184766    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv" event={"ID":"da0e060a-39a4-4b3e-939d-555a1d40122e","Type":"ContainerDied","Data":"8e0af14b0b29921be3ec444c8aa19077f2a16ada380d95ac6500b66309589b42"}
Oct 30 21:57:58.186529 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:58.186500    4882 generic.go:356] "Generic (PLEG): container finished" podID="818304b9-ec45-4dab-99cb-3b9b9df2e62b" containerID="22e216e6c40adc0c9b51daa762d7bf712dc2f3d8b471749338d811e307244e6d" exitCode=0
Oct 30 21:57:58.186715 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:58.186568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb" event={"ID":"818304b9-ec45-4dab-99cb-3b9b9df2e62b","Type":"ContainerDied","Data":"22e216e6c40adc0c9b51daa762d7bf712dc2f3d8b471749338d811e307244e6d"}
Oct 30 21:57:58.186715 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:58.186634    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb" event={"ID":"818304b9-ec45-4dab-99cb-3b9b9df2e62b","Type":"ContainerStarted","Data":"0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64"}
Oct 30 21:57:59.372500 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.372472    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:57:59.375843 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.375818    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:57:59.498809 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.498725    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\") pod \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") "
Oct 30 21:57:59.498809 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.498767    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\") pod \"da0e060a-39a4-4b3e-939d-555a1d40122e\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") "
Oct 30 21:57:59.499045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.498867    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\" (UniqueName: \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\") pod \"da0e060a-39a4-4b3e-939d-555a1d40122e\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") "
Oct 30 21:57:59.499045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.498936    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\" (UniqueName: \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\") pod \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") "
Oct 30 21:57:59.499045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499010    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\") pod \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\" (UID: \"818304b9-ec45-4dab-99cb-3b9b9df2e62b\") "
Oct 30 21:57:59.499207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499065    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\") pod \"da0e060a-39a4-4b3e-939d-555a1d40122e\" (UID: \"da0e060a-39a4-4b3e-939d-555a1d40122e\") "
Oct 30 21:57:59.499207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499195    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca" pod "da0e060a-39a4-4b3e-939d-555a1d40122e" (UID: "da0e060a-39a4-4b3e-939d-555a1d40122e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:59.499633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499568    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca" pod "818304b9-ec45-4dab-99cb-3b9b9df2e62b" (UID: "818304b9-ec45-4dab-99cb-3b9b9df2e62b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:57:59.499835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499775    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.499835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.499797    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.501520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.501495    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca" pod "818304b9-ec45-4dab-99cb-3b9b9df2e62b" (UID: "818304b9-ec45-4dab-99cb-3b9b9df2e62b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:59.501520 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.501500    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn" pod "818304b9-ec45-4dab-99cb-3b9b9df2e62b" (UID: "818304b9-ec45-4dab-99cb-3b9b9df2e62b"). InnerVolumeSpecName "kube-api-access-2jwjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:59.501693 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.501606    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca" pod "da0e060a-39a4-4b3e-939d-555a1d40122e" (UID: "da0e060a-39a4-4b3e-939d-555a1d40122e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:57:59.501796 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.501778    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq" pod "da0e060a-39a4-4b3e-939d-555a1d40122e" (UID: "da0e060a-39a4-4b3e-939d-555a1d40122e"). InnerVolumeSpecName "kube-api-access-skxjq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:57:59.600822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.600782    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da0e060a-39a4-4b3e-939d-555a1d40122e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.600822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.600817    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/818304b9-ec45-4dab-99cb-3b9b9df2e62b-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.600822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.600830    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skxjq\" (UniqueName: \"kubernetes.io/projected/da0e060a-39a4-4b3e-939d-555a1d40122e-kube-api-access-skxjq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:57:59.601065 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:57:59.600843    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jwjn\" (UniqueName: \"kubernetes.io/projected/818304b9-ec45-4dab-99cb-3b9b9df2e62b-kube-api-access-2jwjn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:00.201252 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.201220    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"
Oct 30 21:58:00.201436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.201218    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv" event={"ID":"da0e060a-39a4-4b3e-939d-555a1d40122e","Type":"ContainerDied","Data":"9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f"}
Oct 30 21:58:00.201436 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.201386    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9ab0a92dca752793427cacd9a3ab9b555d349094c8f3a747e4f2eee32a465d0f"
Oct 30 21:58:00.202804 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.202778    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb" event={"ID":"818304b9-ec45-4dab-99cb-3b9b9df2e62b","Type":"ContainerDied","Data":"0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64"}
Oct 30 21:58:00.202925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.202810    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"
Oct 30 21:58:00.202925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:00.202821    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bb8309f7767951be9ae7deac6b282b883cd1cff0175f1bf0a7f8bbef47a0e64"
Oct 30 21:58:03.370277 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:03.370242    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:58:11.035725 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:11.035688    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:58:12.872305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.872263    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"]
Oct 30 21:58:12.872988 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.872968    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da0e060a-39a4-4b3e-939d-555a1d40122e" containerName="revoke-keys"
Oct 30 21:58:12.873069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.872992    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0e060a-39a4-4b3e-939d-555a1d40122e" containerName="revoke-keys"
Oct 30 21:58:12.873069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.873037    4882 cpu_manager.go:414] "containerMap: removing container" podUID="818304b9-ec45-4dab-99cb-3b9b9df2e62b" containerName="revoke-keys"
Oct 30 21:58:12.873069 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.873046    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="818304b9-ec45-4dab-99cb-3b9b9df2e62b" containerName="revoke-keys"
Oct 30 21:58:12.873215 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.873201    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0e060a-39a4-4b3e-939d-555a1d40122e" containerName="revoke-keys"
Oct 30 21:58:12.873257 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.873216    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="818304b9-ec45-4dab-99cb-3b9b9df2e62b" containerName="revoke-keys"
Oct 30 21:58:12.877664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.877646    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:12.931666 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.931626    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"]
Oct 30 21:58:12.973933 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.973896    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:12.973933 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.973937    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:12.974144 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.974042    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cllwl\" (UniqueName: \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:12.978572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.978549    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:12.978703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:12.978577    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:58:13.075217 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.075180    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.075379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.075223    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.075379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.075280    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cllwl\" (UniqueName: \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.075958 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.075938    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.077691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.077669    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.137188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.137110    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cllwl\" (UniqueName: \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\") pod \"maas-api-revoke-sub-41fff10f-qbhnf\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.191648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.191573    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:13.381713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.381671    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"]
Oct 30 21:58:13.386675 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.386646    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.441689 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.441652    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"]
Oct 30 21:58:13.480733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.480694    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cvx\" (UniqueName: \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.480900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.480878    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.480961 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.480937    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.582603 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.582564    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cvx\" (UniqueName: \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.582792 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.582739    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.582792 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.582786    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.583516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.583497    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.585311 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.585287    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.611163 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:13.611128    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdc6bd8a_7ef2_4373_900b_f4b5125f99aa.slice/crio-a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf WatchSource:0}: Error finding container a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf: Status 404 returned error can't find the container with id a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf
Oct 30 21:58:13.614119 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.614096    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"]
Oct 30 21:58:13.644744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.644718    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cvx\" (UniqueName: \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\") pod \"maas-api-revoke-sub-e38c7365-ntpd4\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:13.703274 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:13.703194    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:14.075288 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:14.075258    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79d69ded_65ac_4a92_aceb_7f12baac7c0d.slice/crio-78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1 WatchSource:0}: Error finding container 78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1: Status 404 returned error can't find the container with id 78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1
Oct 30 21:58:14.078649 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:14.078624    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"]
Oct 30 21:58:14.343733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:14.343694    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4" event={"ID":"79d69ded-65ac-4a92-aceb-7f12baac7c0d","Type":"ContainerStarted","Data":"78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1"}
Oct 30 21:58:14.345335 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:14.345309    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf" event={"ID":"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa","Type":"ContainerStarted","Data":"21c7c27ba8cc4b54d3e9bca7c5f95c757836a15870ef0a623b3bbcd041d45817"}
Oct 30 21:58:14.345467 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:14.345360    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf" event={"ID":"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa","Type":"ContainerStarted","Data":"a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf"}
Oct 30 21:58:14.462303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:14.462245    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf" podStartSLOduration=2.462232083 podStartE2EDuration="2.462232083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:14.461664409 +0000 UTC m=+1217.634988836" watchObservedRunningTime="2026-10-30 21:58:14.462232083 +0000 UTC m=+1217.635556510"
Oct 30 21:58:15.354045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:15.354009    4882 generic.go:356] "Generic (PLEG): container finished" podID="79d69ded-65ac-4a92-aceb-7f12baac7c0d" containerID="fce362c244974a0efe4754641d5d4d572e5716b0be00025c684fddc51667bb2e" exitCode=0
Oct 30 21:58:15.354463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:15.354093    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4" event={"ID":"79d69ded-65ac-4a92-aceb-7f12baac7c0d","Type":"ContainerDied","Data":"fce362c244974a0efe4754641d5d4d572e5716b0be00025c684fddc51667bb2e"}
Oct 30 21:58:15.355762 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:15.355731    4882 generic.go:356] "Generic (PLEG): container finished" podID="cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" containerID="21c7c27ba8cc4b54d3e9bca7c5f95c757836a15870ef0a623b3bbcd041d45817" exitCode=0
Oct 30 21:58:15.355893 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:15.355814    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf" event={"ID":"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa","Type":"ContainerDied","Data":"21c7c27ba8cc4b54d3e9bca7c5f95c757836a15870ef0a623b3bbcd041d45817"}
Oct 30 21:58:16.036271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.036227    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" podUID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" containerName="payload-processing" containerID="cri-o://449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c" gracePeriod=25
Oct 30 21:58:16.301957 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.301887    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:58:16.362725 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.362688    4882 generic.go:356] "Generic (PLEG): container finished" podID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" containerID="449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c" exitCode=0
Oct 30 21:58:16.363141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.362734    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn"
Oct 30 21:58:16.363141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.362769    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" event={"ID":"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2","Type":"ContainerDied","Data":"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"}
Oct 30 21:58:16.363141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.362823    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="models-as-a-service/payload-processing-external-model-f854549f8-rptzn" event={"ID":"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2","Type":"ContainerDied","Data":"f7499078fa9ea2ddd92100bff6a6336e9d35efa2bc139e920fcd4dc0cc97117a"}
Oct 30 21:58:16.363141 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.362844    4882 scope.go:122] "RemoveContainer" containerID="449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"
Oct 30 21:58:16.372648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.372628    4882 scope.go:122] "RemoveContainer" containerID="449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"
Oct 30 21:58:16.373018 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:58:16.372991    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c\": container with ID starting with 449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c not found: ID does not exist" containerID="449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"
Oct 30 21:58:16.373100 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.373028    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c"} err="failed to get container status \"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c\": rpc error: code = NotFound desc = could not find container \"449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c\": container with ID starting with 449f821aa6c422e0de0cb1b0f473d50a424ea513a9430f2bcb4aaf11c1f1822c not found: ID does not exist"
Oct 30 21:58:16.414201 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414165    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\") pod \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") "
Oct 30 21:58:16.414392 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\") pod \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") "
Oct 30 21:58:16.414392 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414353    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\") pod \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") "
Oct 30 21:58:16.414521 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414445    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\") pod \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\" (UID: \"197457cc-b2ea-47fb-8acf-b8f4a84c4ce2\") "
Oct 30 21:58:16.414733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414654    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume" pod "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:16.414733 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414663    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume" pod "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2"). InnerVolumeSpecName "routing-overlay-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:16.415002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.414982    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.415060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.415011    4882 reconciler_common.go:299] "Volume detached for volume \"routing-overlay-volume\" (UniqueName: \"kubernetes.io/configmap/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-routing-overlay-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.416969 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.416942    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials" pod "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2"). InnerVolumeSpecName "provider-credentials". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:16.418095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.417929    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6" pod "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" (UID: "197457cc-b2ea-47fb-8acf-b8f4a84c4ce2"). InnerVolumeSpecName "kube-api-access-f92n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:16.516106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.516078    4882 reconciler_common.go:299] "Volume detached for volume \"provider-credentials\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-provider-credentials\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.516106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.516106    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f92n6\" (UniqueName: \"kubernetes.io/projected/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2-kube-api-access-f92n6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.525548 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.525519    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:16.554700 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.554637    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:16.616805 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.616772    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\") pod \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") "
Oct 30 21:58:16.617014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.616832    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\" (UniqueName: \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\") pod \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") "
Oct 30 21:58:16.617014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.616911    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\" (UniqueName: \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\") pod \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") "
Oct 30 21:58:16.617014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.616932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\") pod \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") "
Oct 30 21:58:16.617014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.617004    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\") pod \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\" (UID: \"79d69ded-65ac-4a92-aceb-7f12baac7c0d\") "
Oct 30 21:58:16.617241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.617033    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\") pod \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\" (UID: \"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa\") "
Oct 30 21:58:16.617533 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.617501    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca" pod "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" (UID: "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:16.617680 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.617659    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca" pod "79d69ded-65ac-4a92-aceb-7f12baac7c0d" (UID: "79d69ded-65ac-4a92-aceb-7f12baac7c0d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:16.619206 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.619183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl" pod "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" (UID: "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa"). InnerVolumeSpecName "kube-api-access-cllwl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:16.619295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.619219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca" pod "79d69ded-65ac-4a92-aceb-7f12baac7c0d" (UID: "79d69ded-65ac-4a92-aceb-7f12baac7c0d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:16.619295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.619244    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx" pod "79d69ded-65ac-4a92-aceb-7f12baac7c0d" (UID: "79d69ded-65ac-4a92-aceb-7f12baac7c0d"). InnerVolumeSpecName "kube-api-access-w6cvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:16.619389 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.619361    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca" pod "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" (UID: "cdc6bd8a-7ef2-4373-900b-f4b5125f99aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:16.718286 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718252    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.718286 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718282    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cllwl\" (UniqueName: \"kubernetes.io/projected/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-kube-api-access-cllwl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.718286 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718292    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cvx\" (UniqueName: \"kubernetes.io/projected/79d69ded-65ac-4a92-aceb-7f12baac7c0d-kube-api-access-w6cvx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.718518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718303    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.718518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718314    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79d69ded-65ac-4a92-aceb-7f12baac7c0d-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.718518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.718326    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:16.883953 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.883892    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:58:16.887867 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.887838    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["models-as-a-service/payload-processing-external-model-f854549f8-rptzn"]
Oct 30 21:58:16.943267 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:16.943233    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" path="/var/lib/kubelet/pods/197457cc-b2ea-47fb-8acf-b8f4a84c4ce2/volumes"
Oct 30 21:58:17.377722 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.377664    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4" event={"ID":"79d69ded-65ac-4a92-aceb-7f12baac7c0d","Type":"ContainerDied","Data":"78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1"}
Oct 30 21:58:17.377722 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.377697    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"
Oct 30 21:58:17.378285 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.377809    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78fb5d9fd799b262613e6c816e6ca3959902b8e6334cfa72c0e74401e7d080d1"
Oct 30 21:58:17.380419 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.380395    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"
Oct 30 21:58:17.380561 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.380414    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf" event={"ID":"cdc6bd8a-7ef2-4373-900b-f4b5125f99aa","Type":"ContainerDied","Data":"a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf"}
Oct 30 21:58:17.380561 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:17.380460    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a59edb5ab9a797c68da58046716039c25bfb06eff1bde3f4930fa996e322cfbf"
Oct 30 21:58:20.806768 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:58:20.806719    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:58:28.771306 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771266    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 21:58:28.771783 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771768    4882 cpu_manager.go:414] "containerMap: removing container" podUID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" containerName="payload-processing"
Oct 30 21:58:28.771823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771784    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" containerName="payload-processing"
Oct 30 21:58:28.771823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771801    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" containerName="revoke-keys"
Oct 30 21:58:28.771823 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771806    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" containerName="revoke-keys"
Oct 30 21:58:28.771941 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771835    4882 cpu_manager.go:414] "containerMap: removing container" podUID="79d69ded-65ac-4a92-aceb-7f12baac7c0d" containerName="revoke-keys"
Oct 30 21:58:28.771941 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771842    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79d69ded-65ac-4a92-aceb-7f12baac7c0d" containerName="revoke-keys"
Oct 30 21:58:28.771941 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771917    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="79d69ded-65ac-4a92-aceb-7f12baac7c0d" containerName="revoke-keys"
Oct 30 21:58:28.771941 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771929    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="197457cc-b2ea-47fb-8acf-b8f4a84c4ce2" containerName="payload-processing"
Oct 30 21:58:28.771941 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.771936    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" containerName="revoke-keys"
Oct 30 21:58:28.776744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.776723    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.832202 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.832167    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9a5623-openshift-default-dockercfg-gltcf\""
Oct 30 21:58:28.885995 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.885958    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 21:58:28.909199 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909160    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909376 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909214    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909376 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909241    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909455 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cfhd\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909513 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909644    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909717    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:28.909861 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:28.909844    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.010833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.010796    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.010863    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.010908    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.010941    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.010987    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011040    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011114    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cfhd\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011145    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011265    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011330    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011500    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011687 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011666    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.011824 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.011670    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.013494 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.013447    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.014008 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.013980    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.046001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.045918    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:58:29.077921 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.077886    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cfhd\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.078518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.078487    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.090551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.090511    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:29.526359 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.526321    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:29.526498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.526397    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:29.526498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.526465    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:29.527454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:29.527426    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 21:58:30.017091 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.017054    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"]
Oct 30 21:58:30.021118 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.021084    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.078371 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.078340    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:30.080337 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.080308    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"]
Oct 30 21:58:30.124054 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.124021    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.124234 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.124172    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnvqc\" (UniqueName: \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.124416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.124398    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.129225 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.129194    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:58:30.133616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.131559    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"]
Oct 30 21:58:30.139124 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.139096    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.160728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.160693    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"]
Oct 30 21:58:30.239340 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.239298    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.240162 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.240130    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnvqc\" (UniqueName: \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.247616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.240356    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.248035 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.244747    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.255618 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.255504    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.307253 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.307177    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnvqc\" (UniqueName: \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\") pod \"maas-api-revoke-sub-e38c6beb-kqc29\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.345611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.344785    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:30.350257 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.350222    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.350459 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.350294    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbwq\" (UniqueName: \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.350529 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.350454    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.451638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.451337    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.451638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.451423    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.451638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.451479    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbwq\" (UniqueName: \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.452979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.452922    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.455737 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.455576    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.513525 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.513493    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbwq\" (UniqueName: \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\") pod \"maas-api-revoke-sub-d0ae0b47-k4z56\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:30.556838 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.556795    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" event={"ID":"77d609a1-ec75-4947-ae8a-588901300e7b","Type":"ContainerStarted","Data":"4cdcc51a2a4ee22ce7dbcbf89e847fd92bb24ac602c9792b2f1976dd60af364b"}
Oct 30 21:58:30.557035 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.556859    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" event={"ID":"77d609a1-ec75-4947-ae8a-588901300e7b","Type":"ContainerStarted","Data":"683eae707c85213123f4bc2c5b28a843ac50b0bd5468aab4c2754a734a05c64d"}
Oct 30 21:58:30.680221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.680164    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podStartSLOduration=2.680148589 podStartE2EDuration="2.680148589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:30.677827278 +0000 UTC m=+1233.851151706" watchObservedRunningTime="2026-10-30 21:58:30.680148589 +0000 UTC m=+1233.853473074"
Oct 30 21:58:30.722631 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:30.722578    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod066a1cd2_b49e_40a3_84c1_b5ac0b9b739f.slice/crio-e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092 WatchSource:0}: Error finding container e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092: Status 404 returned error can't find the container with id e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092
Oct 30 21:58:30.749611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.748947    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"]
Oct 30 21:58:30.771073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:30.771045    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:31.091530 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.091496    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:31.092972 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.092947    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:31.093093 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.093007    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:31.174837 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:31.174805    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a2b8645_1bc6_4bf3_b016_008f09b2dc8f.slice/crio-7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578 WatchSource:0}: Error finding container 7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578: Status 404 returned error can't find the container with id 7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578
Oct 30 21:58:31.179303 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.179274    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"]
Oct 30 21:58:31.566269 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.566238    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56" event={"ID":"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f","Type":"ContainerStarted","Data":"6701458106faeab96d7386a99d0223af998941d1b44530d18802673cfef9d4a4"}
Oct 30 21:58:31.566417 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.566291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56" event={"ID":"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f","Type":"ContainerStarted","Data":"7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578"}
Oct 30 21:58:31.567883 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.567852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29" event={"ID":"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f","Type":"ContainerStarted","Data":"5ce2302a67737d0021dd6776afb3416cfc3fc1a38fee2f1674ab0e26e46a5d94"}
Oct 30 21:58:31.568031 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.567897    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29" event={"ID":"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f","Type":"ContainerStarted","Data":"e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092"}
Oct 30 21:58:31.685711 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.685661    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56" podStartSLOduration=1.685646475 podStartE2EDuration="1.685646475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:31.682026272 +0000 UTC m=+1234.855350699" watchObservedRunningTime="2026-10-30 21:58:31.685646475 +0000 UTC m=+1234.858970902"
Oct 30 21:58:31.815736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:31.815633    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29" podStartSLOduration=2.815618578 podStartE2EDuration="2.815618578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:31.813784557 +0000 UTC m=+1234.987108984" watchObservedRunningTime="2026-10-30 21:58:31.815618578 +0000 UTC m=+1234.988943005"
Oct 30 21:58:32.090952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.090918    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:32.091107 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.090993    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:32.575114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.575022    4882 generic.go:356] "Generic (PLEG): container finished" podID="2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" containerID="6701458106faeab96d7386a99d0223af998941d1b44530d18802673cfef9d4a4" exitCode=0
Oct 30 21:58:32.575544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.575108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56" event={"ID":"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f","Type":"ContainerDied","Data":"6701458106faeab96d7386a99d0223af998941d1b44530d18802673cfef9d4a4"}
Oct 30 21:58:32.576817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.576791    4882 generic.go:356] "Generic (PLEG): container finished" podID="066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" containerID="5ce2302a67737d0021dd6776afb3416cfc3fc1a38fee2f1674ab0e26e46a5d94" exitCode=0
Oct 30 21:58:32.576937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:32.576875    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29" event={"ID":"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f","Type":"ContainerDied","Data":"5ce2302a67737d0021dd6776afb3416cfc3fc1a38fee2f1674ab0e26e46a5d94"}
Oct 30 21:58:33.091749 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.091708    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:33.091937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.091777    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:33.800967 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.800934    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:33.804459 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.804430    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:33.894529 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894432    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\") pod \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") "
Oct 30 21:58:33.894744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894567    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\") pod \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") "
Oct 30 21:58:33.894744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894662    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\") pod \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") "
Oct 30 21:58:33.894871 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894747    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\" (UniqueName: \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\") pod \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\" (UID: \"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f\") "
Oct 30 21:58:33.894871 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894810    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\") pod \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") "
Oct 30 21:58:33.894977 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894890    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\" (UniqueName: \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\") pod \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\" (UID: \"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f\") "
Oct 30 21:58:33.894977 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.894922    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca" pod "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" (UID: "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:33.895291 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.895261    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca" pod "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" (UID: "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:33.895482 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.895461    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.895556 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.895487    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:33.896908 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.896890    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca" pod "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" (UID: "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:33.897028 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.896909    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca" pod "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" (UID: "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:33.897262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.897242    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq" pod "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" (UID: "2a2b8645-1bc6-4bf3-b016-008f09b2dc8f"). InnerVolumeSpecName "kube-api-access-5kbwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:33.897262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.897253    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc" pod "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" (UID: "066a1cd2-b49e-40a3-84c1-b5ac0b9b739f"). InnerVolumeSpecName "kube-api-access-pnvqc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:34.000619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.997652    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:34.000619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.997763    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnvqc\" (UniqueName: \"kubernetes.io/projected/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-kube-api-access-pnvqc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:34.000619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.997783    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbwq\" (UniqueName: \"kubernetes.io/projected/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f-kube-api-access-5kbwq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:34.000619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:33.997801    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:34.091896 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.091853    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:34.092098 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.091920    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:34.590565 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.590525    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56" event={"ID":"2a2b8645-1bc6-4bf3-b016-008f09b2dc8f","Type":"ContainerDied","Data":"7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578"}
Oct 30 21:58:34.590753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.590660    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c7aa5975d57a1a29ce13beffa68d7c43ff4e52bceb753728f82f34c0f3cc578"
Oct 30 21:58:34.590753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.590734    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"
Oct 30 21:58:34.595891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.595865    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"
Oct 30 21:58:34.596015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.595883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29" event={"ID":"066a1cd2-b49e-40a3-84c1-b5ac0b9b739f","Type":"ContainerDied","Data":"e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092"}
Oct 30 21:58:34.596015 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:34.595943    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2cab6ca3d6e9735a7f3e1dd463ed08834688e17a2039e66452c536cfcbe8092"
Oct 30 21:58:35.091671 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:35.091638    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:35.092039 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:35.091691    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:36.091463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:36.091425    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Oct 30 21:58:36.091677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:36.091507    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Oct 30 21:58:37.096190 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:37.096160    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:37.096566 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:37.096453    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:37.097134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:37.097118    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 21:58:38.378372 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.378341    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:58:38.898002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.897966    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 21:58:38.898459 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898446    4882 cpu_manager.go:414] "containerMap: removing container" podUID="066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" containerName="revoke-keys"
Oct 30 21:58:38.898505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898461    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" containerName="revoke-keys"
Oct 30 21:58:38.898505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898475    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" containerName="revoke-keys"
Oct 30 21:58:38.898505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898480    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" containerName="revoke-keys"
Oct 30 21:58:38.898628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898558    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" containerName="revoke-keys"
Oct 30 21:58:38.898628 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.898574    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" containerName="revoke-keys"
Oct 30 21:58:38.903060 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.903035    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.952920 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.952875    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.952971    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953120    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953223 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953188    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953294 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953237    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953340 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953386 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953369    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9lr5\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.953428 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.953416    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:38.956277 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:38.956258    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-56f3fe-openshift-default-dockercfg-48wbx\""
Oct 30 21:58:39.011234 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.011192    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 21:58:39.054540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054500    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.054540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054543    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9lr5\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.054813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054582    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.054813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054647    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.054813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054714    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.054813 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054764    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055017 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054819    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055017 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054890    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055017 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054922    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055017 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.054980    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055218 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.055058    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055267 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.055252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055489 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.055464    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.055640 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.055619    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.057786 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.057749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.057904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.057821    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.116971 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.116923    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9lr5\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.117487 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.117455    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.217528 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.217333    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:39.651307 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.651264    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 21:58:39.682337 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:39.682300    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53fcd401_e36f_45c3_b3c5_1af35be22bb7.slice/crio-855e0a3ae8e77cf3bc5e953cfdbe782be6a6f95b3b841cff631070d0944f1636 WatchSource:0}: Error finding container 855e0a3ae8e77cf3bc5e953cfdbe782be6a6f95b3b841cff631070d0944f1636: Status 404 returned error can't find the container with id 855e0a3ae8e77cf3bc5e953cfdbe782be6a6f95b3b841cff631070d0944f1636
Oct 30 21:58:39.685752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.685716    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:39.685890 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.685800    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:39.685890 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:39.685840    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:40.074925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.072154    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 21:58:40.092122 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.090791    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.137089 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.137041    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 21:58:40.196612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.196562    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4d14e0-openshift-default-dockercfg-mqz9f\""
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.205929    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.205984    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48smv\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206039    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206111    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206128    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206212    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.208651 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.206264    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308022 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.307976    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308045    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48smv\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308119    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308147    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308174    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308493 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308258    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308493 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308398    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.308703 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.308672    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.309057 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.309027    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.309150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.309094    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.309150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.309125    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.309271 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.309196    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.311269 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.311247    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.311859 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.311831    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.372789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.372749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-48smv\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.372964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.372944    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.412430 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.412388    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:40.658665 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.658558    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" event={"ID":"53fcd401-e36f-45c3-b3c5-1af35be22bb7","Type":"ContainerStarted","Data":"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"}
Oct 30 21:58:40.658665 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.658635    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" event={"ID":"53fcd401-e36f-45c3-b3c5-1af35be22bb7","Type":"ContainerStarted","Data":"855e0a3ae8e77cf3bc5e953cfdbe782be6a6f95b3b841cff631070d0944f1636"}
Oct 30 21:58:40.785473 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.785405    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podStartSLOduration=2.785383491 podStartE2EDuration="2.785383491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:40.780345622 +0000 UTC m=+1243.953670049" watchObservedRunningTime="2026-10-30 21:58:40.785383491 +0000 UTC m=+1243.958707919"
Oct 30 21:58:40.858494 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:40.858458    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4fd6f23_4b8c_4104_8fe8_20d19f9d7d9e.slice/crio-45b42e5618b3a2bbcecbae2d6ab63ff6770558f4f4877506dea0a6fdc502c571 WatchSource:0}: Error finding container 45b42e5618b3a2bbcecbae2d6ab63ff6770558f4f4877506dea0a6fdc502c571: Status 404 returned error can't find the container with id 45b42e5618b3a2bbcecbae2d6ab63ff6770558f4f4877506dea0a6fdc502c571
Oct 30 21:58:40.860387 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.860363    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 21:58:40.861099 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.861069    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:40.861161 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.861133    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:40.861203 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:40.861160    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:58:41.023614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.023557    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"]
Oct 30 21:58:41.039611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.037934    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.086970 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.084939    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"]
Oct 30 21:58:41.091533 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.091500    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:58:41.091707 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.091580    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:41.117379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.117341    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.117544 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.117404    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.117626 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.117602    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm5bn\" (UniqueName: \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.217622 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.217513    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:41.219021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.218994    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.219106 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.219044    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.219187 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.219173    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm5bn\" (UniqueName: \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.219369 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.219331    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:41.219450 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.219386    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:41.219773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.219749    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.245614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.243171    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.283745 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.283710    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm5bn\" (UniqueName: \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\") pod \"maas-api-revoke-sub-cd157ae1-4tmxn\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.389773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.389733    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:41.681609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.681413    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" event={"ID":"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e","Type":"ContainerStarted","Data":"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"}
Oct 30 21:58:41.681609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.681481    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" event={"ID":"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e","Type":"ContainerStarted","Data":"45b42e5618b3a2bbcecbae2d6ab63ff6770558f4f4877506dea0a6fdc502c571"}
Oct 30 21:58:41.797931 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:41.797881    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd45007a3_1435_458c_b3b3_288f2345b939.slice/crio-af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9 WatchSource:0}: Error finding container af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9: Status 404 returned error can't find the container with id af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9
Oct 30 21:58:41.801440 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.801420    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"]
Oct 30 21:58:41.811617 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:41.811549    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podStartSLOduration=1.811531031 podStartE2EDuration="1.811531031s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:41.808309983 +0000 UTC m=+1244.981634434" watchObservedRunningTime="2026-10-30 21:58:41.811531031 +0000 UTC m=+1244.984855460"
Oct 30 21:58:42.217747 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.217716    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.217924 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.217764    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:42.413020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.412989    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:42.414624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.414577    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:42.414833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.414813    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:42.688677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.688642    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn" event={"ID":"d45007a3-1435-458c-b3b3-288f2345b939","Type":"ContainerStarted","Data":"7e2a14d1035ef3aaab5b63307521fc3161eaeffaf7638616984b941562c61faa"}
Oct 30 21:58:42.689086 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.688698    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn" event={"ID":"d45007a3-1435-458c-b3b3-288f2345b939","Type":"ContainerStarted","Data":"af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9"}
Oct 30 21:58:42.809846 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:42.809451    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn" podStartSLOduration=2.8094352000000002 podStartE2EDuration="2.8094352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:42.807817495 +0000 UTC m=+1245.981141922" watchObservedRunningTime="2026-10-30 21:58:42.8094352 +0000 UTC m=+1245.982759671"
Oct 30 21:58:43.218105 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.218074    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.218353 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.218329    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:43.413444 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.413407    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:43.413661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.413456    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:43.696664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.695581    4882 generic.go:356] "Generic (PLEG): container finished" podID="d45007a3-1435-458c-b3b3-288f2345b939" containerID="7e2a14d1035ef3aaab5b63307521fc3161eaeffaf7638616984b941562c61faa" exitCode=0
Oct 30 21:58:43.696664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:43.696534    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn" event={"ID":"d45007a3-1435-458c-b3b3-288f2345b939","Type":"ContainerDied","Data":"7e2a14d1035ef3aaab5b63307521fc3161eaeffaf7638616984b941562c61faa"}
Oct 30 21:58:44.077055 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.076940    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerName="main" containerID="cri-o://0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0" gracePeriod=45
Oct 30 21:58:44.186183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.186140    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerName="main" probeResult="failure" output="Get \"https://10.132.0.60:8000/ready\": dial tcp 10.132.0.60:8000: connect: connection refused"
Oct 30 21:58:44.217865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.217831    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.218034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.217891    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:44.412934 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.412898    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:44.413132 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.412955    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:44.520580 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.520552    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618219    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618289    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618329    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618422    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618468    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\" (UniqueName: \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.619551 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.618654    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\") pod \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\" (UID: \"e6a8b679-1a6e-4ccc-ab8b-576055e6843b\") "
Oct 30 21:58:44.620242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.620046    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:44.620524 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.620369    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:44.620806 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.620695    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:44.623512 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.623472    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 21:58:44.623639 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.623504    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:44.623639 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.623616    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg" pod "e6a8b679-1a6e-4ccc-ab8b-576055e6843b" (UID: "e6a8b679-1a6e-4ccc-ab8b-576055e6843b"). InnerVolumeSpecName "kube-api-access-5hrpg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768902    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768942    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hrpg\" (UniqueName: \"kubernetes.io/projected/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-kube-api-access-5hrpg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768959    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768972    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768986    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.770692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.768999    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6a8b679-1a6e-4ccc-ab8b-576055e6843b-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:44.803663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.778458    4882 generic.go:356] "Generic (PLEG): container finished" podID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerID="0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0" exitCode=0
Oct 30 21:58:44.803663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.799736    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"
Oct 30 21:58:44.809856 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.809740    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" event={"ID":"e6a8b679-1a6e-4ccc-ab8b-576055e6843b","Type":"ContainerDied","Data":"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"}
Oct 30 21:58:44.810125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.809831    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9" event={"ID":"e6a8b679-1a6e-4ccc-ab8b-576055e6843b","Type":"ContainerDied","Data":"14aa6a89ea9bd41c6c0047a6143523ec12c544503a6185d81d5a54fc7e9ae423"}
Oct 30 21:58:44.814633 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.810103    4882 scope.go:122] "RemoveContainer" containerID="0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"
Oct 30 21:58:44.836134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.835760    4882 scope.go:122] "RemoveContainer" containerID="0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"
Oct 30 21:58:44.836518 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:58:44.836412    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0\": container with ID starting with 0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0 not found: ID does not exist" containerID="0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"
Oct 30 21:58:44.836518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.836457    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0"} err="failed to get container status \"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0\": rpc error: code = NotFound desc = could not find container \"0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0\": container with ID starting with 0db1d5fba33e428cfffcb61e49352cd47e338d9fea4b5922723aa518282c2ca0 not found: ID does not exist"
Oct 30 21:58:44.993077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.993046    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:58:44.998399 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:44.998372    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/e2e-conflict-a-09506444-kserve-5f65586799-bv8r9"]
Oct 30 21:58:45.086661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.083368    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"]
Oct 30 21:58:45.086661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.083429    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95b299a2-f8wgw"]
Oct 30 21:58:45.119514 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.119484    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:45.218028 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.217987    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.218209 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.218040    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:45.294974 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.294932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\") pod \"d45007a3-1435-458c-b3b3-288f2345b939\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") "
Oct 30 21:58:45.294974 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.294976    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\" (UniqueName: \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\") pod \"d45007a3-1435-458c-b3b3-288f2345b939\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") "
Oct 30 21:58:45.295239 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.295220    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\") pod \"d45007a3-1435-458c-b3b3-288f2345b939\" (UID: \"d45007a3-1435-458c-b3b3-288f2345b939\") "
Oct 30 21:58:45.295816 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.295784    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca" pod "d45007a3-1435-458c-b3b3-288f2345b939" (UID: "d45007a3-1435-458c-b3b3-288f2345b939"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:45.297444 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.297414    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn" pod "d45007a3-1435-458c-b3b3-288f2345b939" (UID: "d45007a3-1435-458c-b3b3-288f2345b939"). InnerVolumeSpecName "kube-api-access-bm5bn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:45.297543 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.297419    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca" pod "d45007a3-1435-458c-b3b3-288f2345b939" (UID: "d45007a3-1435-458c-b3b3-288f2345b939"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:45.396950 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.396872    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:45.397118 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.397105    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d45007a3-1435-458c-b3b3-288f2345b939-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:45.397187 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.397179    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm5bn\" (UniqueName: \"kubernetes.io/projected/d45007a3-1435-458c-b3b3-288f2345b939-kube-api-access-bm5bn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:45.413794 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.413758    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:45.413952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.413812    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:45.787925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.787843    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"
Oct 30 21:58:45.787925 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.787908    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn" event={"ID":"d45007a3-1435-458c-b3b3-288f2345b939","Type":"ContainerDied","Data":"af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9"}
Oct 30 21:58:45.788325 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:45.787959    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af97946842266fc6fbd7dc32eb697e00461dd86320f083000fc1e880d58b43f9"
Oct 30 21:58:46.217865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.217827    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused" start-of-body=
Oct 30 21:58:46.218094 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.217905    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": dial tcp 10.132.0.74:15021: connect: connection refused"
Oct 30 21:58:46.413137 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.413100    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:46.413295 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.413168    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:46.960318 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.960283    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="275063ec-f8d8-4fb9-9828-1ccdfb239905" path="/var/lib/kubelet/pods/275063ec-f8d8-4fb9-9828-1ccdfb239905/volumes"
Oct 30 21:58:46.960784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:46.960768    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" path="/var/lib/kubelet/pods/e6a8b679-1a6e-4ccc-ab8b-576055e6843b/volumes"
Oct 30 21:58:47.412811 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:47.412776    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused" start-of-body=
Oct 30 21:58:47.413029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:47.412838    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.75:15021/healthz/ready\": dial tcp 10.132.0.75:15021: connect: connection refused"
Oct 30 21:58:47.487062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:47.487028    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:47.487907 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:47.487884    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:47.488983 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:47.488960    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 21:58:48.386603 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.386562    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"]
Oct 30 21:58:48.387332 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387308    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerName="main"
Oct 30 21:58:48.387421 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387333    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerName="main"
Oct 30 21:58:48.387421 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387348    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d45007a3-1435-458c-b3b3-288f2345b939" containerName="revoke-keys"
Oct 30 21:58:48.387421 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387358    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d45007a3-1435-458c-b3b3-288f2345b939" containerName="revoke-keys"
Oct 30 21:58:48.387603 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387529    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6a8b679-1a6e-4ccc-ab8b-576055e6843b" containerName="main"
Oct 30 21:58:48.387603 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.387544    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d45007a3-1435-458c-b3b3-288f2345b939" containerName="revoke-keys"
Oct 30 21:58:48.391475 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.391408    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.453153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.453104    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"]
Oct 30 21:58:48.482818 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.482789    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:48.483207 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.483174    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:48.484390 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.484367    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 21:58:48.498053 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.498024    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:48.498188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.498138    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:58:48.540572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.540530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.540727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.540653    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.540727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.540693    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdf25\" (UniqueName: \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.641935 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.641839    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.641935 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.641927    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.642163 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.641969    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdf25\" (UniqueName: \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.642959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.642931    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.649629 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.649604    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.704301 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.704273    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdf25\" (UniqueName: \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\") pod \"maas-api-revoke-sub-227baa92-ncbjb\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:48.708076 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:48.708048    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:49.083115 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:49.083080    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d6c496_0bf8_40bb_99ec_e5e028242034.slice/crio-7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e WatchSource:0}: Error finding container 7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e: Status 404 returned error can't find the container with id 7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e
Oct 30 21:58:49.086778 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.086748    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"]
Oct 30 21:58:49.402078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.402044    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"]
Oct 30 21:58:49.405938 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.405917    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.445960 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.445921    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"]
Oct 30 21:58:49.552210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.552177    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.552210 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.552215    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.552471 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.552332    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5nbq\" (UniqueName: \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.653499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.653403    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.653499 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.653454    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.653726 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.653527    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5nbq\" (UniqueName: \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.654333 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.654313    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.656024 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.656003    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.716025 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.715989    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5nbq\" (UniqueName: \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\") pod \"maas-api-revoke-sub-3e04c2c1-q99fd\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.718756 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.718732    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:49.823016 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.822972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb" event={"ID":"32d6c496-0bf8-40bb-99ec-e5e028242034","Type":"ContainerStarted","Data":"8c36478cd3ebf120a49508da64f736e92280e5de4874c640233ffaabacafe876"}
Oct 30 21:58:49.823188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.823051    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb" event={"ID":"32d6c496-0bf8-40bb-99ec-e5e028242034","Type":"ContainerStarted","Data":"7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e"}
Oct 30 21:58:49.942269 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:49.942165    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb" podStartSLOduration=1.942149256 podStartE2EDuration="1.942149256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:49.939279358 +0000 UTC m=+1253.112603785" watchObservedRunningTime="2026-10-30 21:58:49.942149256 +0000 UTC m=+1253.115473714"
Oct 30 21:58:50.096818 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:50.096784    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacea530e_4302_4832_8b70_f3f3ff846808.slice/crio-16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2 WatchSource:0}: Error finding container 16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2: Status 404 returned error can't find the container with id 16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2
Oct 30 21:58:50.099764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.099728    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"]
Oct 30 21:58:50.806746 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:58:50.806706    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:58:50.830743 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.830703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd" event={"ID":"acea530e-4302-4832-8b70-f3f3ff846808","Type":"ContainerStarted","Data":"628995f039ce73e3349f173c67bed560c41f29d265fc5a6bdbc02c97065c5f53"}
Oct 30 21:58:50.830922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.830798    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd" event={"ID":"acea530e-4302-4832-8b70-f3f3ff846808","Type":"ContainerStarted","Data":"16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2"}
Oct 30 21:58:50.832688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.832645    4882 generic.go:356] "Generic (PLEG): container finished" podID="32d6c496-0bf8-40bb-99ec-e5e028242034" containerID="8c36478cd3ebf120a49508da64f736e92280e5de4874c640233ffaabacafe876" exitCode=0
Oct 30 21:58:50.832820 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.832730    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb" event={"ID":"32d6c496-0bf8-40bb-99ec-e5e028242034","Type":"ContainerDied","Data":"8c36478cd3ebf120a49508da64f736e92280e5de4874c640233ffaabacafe876"}
Oct 30 21:58:50.949197 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:50.949091    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd" podStartSLOduration=1.949076989 podStartE2EDuration="1.949076989s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:50.947043776 +0000 UTC m=+1254.120368205" watchObservedRunningTime="2026-10-30 21:58:50.949076989 +0000 UTC m=+1254.122401416"
Oct 30 21:58:51.840153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:51.840113    4882 generic.go:356] "Generic (PLEG): container finished" podID="acea530e-4302-4832-8b70-f3f3ff846808" containerID="628995f039ce73e3349f173c67bed560c41f29d265fc5a6bdbc02c97065c5f53" exitCode=0
Oct 30 21:58:51.840624 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:51.840189    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd" event={"ID":"acea530e-4302-4832-8b70-f3f3ff846808","Type":"ContainerDied","Data":"628995f039ce73e3349f173c67bed560c41f29d265fc5a6bdbc02c97065c5f53"}
Oct 30 21:58:51.988931 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:51.988907    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:52.084264 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.084229    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\" (UniqueName: \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\") pod \"32d6c496-0bf8-40bb-99ec-e5e028242034\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") "
Oct 30 21:58:52.084416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.084349    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\") pod \"32d6c496-0bf8-40bb-99ec-e5e028242034\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") "
Oct 30 21:58:52.084416 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.084392    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\") pod \"32d6c496-0bf8-40bb-99ec-e5e028242034\" (UID: \"32d6c496-0bf8-40bb-99ec-e5e028242034\") "
Oct 30 21:58:52.084838 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.084805    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca" pod "32d6c496-0bf8-40bb-99ec-e5e028242034" (UID: "32d6c496-0bf8-40bb-99ec-e5e028242034"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:52.085078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.085055    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.086511 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.086486    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25" pod "32d6c496-0bf8-40bb-99ec-e5e028242034" (UID: "32d6c496-0bf8-40bb-99ec-e5e028242034"). InnerVolumeSpecName "kube-api-access-kdf25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:52.086511 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.086502    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca" pod "32d6c496-0bf8-40bb-99ec-e5e028242034" (UID: "32d6c496-0bf8-40bb-99ec-e5e028242034"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:52.186761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.186723    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kdf25\" (UniqueName: \"kubernetes.io/projected/32d6c496-0bf8-40bb-99ec-e5e028242034-kube-api-access-kdf25\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.186761 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.186756    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32d6c496-0bf8-40bb-99ec-e5e028242034-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:52.848531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.848497    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"
Oct 30 21:58:52.848531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.848524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb" event={"ID":"32d6c496-0bf8-40bb-99ec-e5e028242034","Type":"ContainerDied","Data":"7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e"}
Oct 30 21:58:52.849001 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:52.848572    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c92d5a25fb420f5ca851c5f688f9fafdec7d64788e01881c1307bab03dee13e"
Oct 30 21:58:53.007620 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.007577    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:53.198077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.197979    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\") pod \"acea530e-4302-4832-8b70-f3f3ff846808\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") "
Oct 30 21:58:53.198077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.198037    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\") pod \"acea530e-4302-4832-8b70-f3f3ff846808\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") "
Oct 30 21:58:53.198077 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.198080    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\" (UniqueName: \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\") pod \"acea530e-4302-4832-8b70-f3f3ff846808\" (UID: \"acea530e-4302-4832-8b70-f3f3ff846808\") "
Oct 30 21:58:53.198456 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.198430    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca" pod "acea530e-4302-4832-8b70-f3f3ff846808" (UID: "acea530e-4302-4832-8b70-f3f3ff846808"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:58:53.198656 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.198576    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:53.200215 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.200191    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq" pod "acea530e-4302-4832-8b70-f3f3ff846808" (UID: "acea530e-4302-4832-8b70-f3f3ff846808"). InnerVolumeSpecName "kube-api-access-w5nbq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:58:53.200293 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.200258    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca" pod "acea530e-4302-4832-8b70-f3f3ff846808" (UID: "acea530e-4302-4832-8b70-f3f3ff846808"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:58:53.300029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.299985    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/acea530e-4302-4832-8b70-f3f3ff846808-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:53.300029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.300019    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5nbq\" (UniqueName: \"kubernetes.io/projected/acea530e-4302-4832-8b70-f3f3ff846808-kube-api-access-w5nbq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:58:53.855742 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.855710    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"
Oct 30 21:58:53.855742 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.855729    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd" event={"ID":"acea530e-4302-4832-8b70-f3f3ff846808","Type":"ContainerDied","Data":"16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2"}
Oct 30 21:58:53.856174 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:53.855784    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="16aec4688243af3d6f8200825b3bb39b29ebbea7fb7661467596cfb5df6f8ca2"
Oct 30 21:58:56.345721 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.345683    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"]
Oct 30 21:58:56.346212 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346156    4882 cpu_manager.go:414] "containerMap: removing container" podUID="32d6c496-0bf8-40bb-99ec-e5e028242034" containerName="revoke-keys"
Oct 30 21:58:56.346212 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346169    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d6c496-0bf8-40bb-99ec-e5e028242034" containerName="revoke-keys"
Oct 30 21:58:56.346212 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346184    4882 cpu_manager.go:414] "containerMap: removing container" podUID="acea530e-4302-4832-8b70-f3f3ff846808" containerName="revoke-keys"
Oct 30 21:58:56.346212 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346190    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acea530e-4302-4832-8b70-f3f3ff846808" containerName="revoke-keys"
Oct 30 21:58:56.346425 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346286    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d6c496-0bf8-40bb-99ec-e5e028242034" containerName="revoke-keys"
Oct 30 21:58:56.346425 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.346295    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="acea530e-4302-4832-8b70-f3f3ff846808" containerName="revoke-keys"
Oct 30 21:58:56.349863 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.349844    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.405188 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.405155    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"]
Oct 30 21:58:56.430619 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.430566    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5p8n\" (UniqueName: \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.430772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.430637    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.430772 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.430743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.451657 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.451622    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:58:56.451832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.451671    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:58:56.531804 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.531767    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5p8n\" (UniqueName: \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.531975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.531840    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.531975 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.531870    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.532723 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.532687    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.534398 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.534374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.593939 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.593910    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5p8n\" (UniqueName: \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\") pod \"maas-api-revoke-sub-125d02ff-79szd\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:56.663332 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:56.663246    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:58:57.039412 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:57.039377    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f4410dd_c22c_4cc4_9632_80166b19c500.slice/crio-8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b WatchSource:0}: Error finding container 8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b: Status 404 returned error can't find the container with id 8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b
Oct 30 21:58:57.042903 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.042871    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"]
Oct 30 21:58:57.214037 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.214003    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"]
Oct 30 21:58:57.219358 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.219323    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.240468 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.240437    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.240653 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.240565    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.240743 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.240725    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqntt\" (UniqueName: \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.287020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.286934    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"]
Oct 30 21:58:57.341834 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.341808    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqntt\" (UniqueName: \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.341969 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.341921    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.342062 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.342047    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.342642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.342564    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.344564 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.344536    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.404002 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.403966    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqntt\" (UniqueName: \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\") pod \"maas-api-revoke-sub-811d30e7-zphn4\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.534741 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.534705    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:58:57.883735 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.883689    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd" event={"ID":"1f4410dd-c22c-4cc4-9632-80166b19c500","Type":"ContainerStarted","Data":"1e211c1ce167cfd6209c6493f32afb5569adab4393472ba8db00e234d2d7f1cb"}
Oct 30 21:58:57.883917 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.883766    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd" event={"ID":"1f4410dd-c22c-4cc4-9632-80166b19c500","Type":"ContainerStarted","Data":"8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b"}
Oct 30 21:58:57.916735 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:58:57.916697    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod750c3524_e7fe_452d_9f22_b0059092ee36.slice/crio-a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b WatchSource:0}: Error finding container a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b: Status 404 returned error can't find the container with id a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b
Oct 30 21:58:57.919964 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:57.919937    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"]
Oct 30 21:58:58.008342 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.008284    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd" podStartSLOduration=2.0082648 podStartE2EDuration="2.0082648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:58:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:58:58.00586393 +0000 UTC m=+1261.179188357" watchObservedRunningTime="2026-10-30 21:58:58.0082648 +0000 UTC m=+1261.181589226"
Oct 30 21:58:58.891987 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.891950    4882 generic.go:356] "Generic (PLEG): container finished" podID="750c3524-e7fe-452d-9f22-b0059092ee36" containerID="3c3f47ab11c8a0613ec0876a0ead75719597a79f54a177c4823911d40ba4b293" exitCode=0
Oct 30 21:58:58.892418 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.892033    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4" event={"ID":"750c3524-e7fe-452d-9f22-b0059092ee36","Type":"ContainerDied","Data":"3c3f47ab11c8a0613ec0876a0ead75719597a79f54a177c4823911d40ba4b293"}
Oct 30 21:58:58.892418 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.892138    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4" event={"ID":"750c3524-e7fe-452d-9f22-b0059092ee36","Type":"ContainerStarted","Data":"a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b"}
Oct 30 21:58:58.893906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.893860    4882 generic.go:356] "Generic (PLEG): container finished" podID="1f4410dd-c22c-4cc4-9632-80166b19c500" containerID="1e211c1ce167cfd6209c6493f32afb5569adab4393472ba8db00e234d2d7f1cb" exitCode=0
Oct 30 21:58:58.894035 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:58:58.893942    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd" event={"ID":"1f4410dd-c22c-4cc4-9632-80166b19c500","Type":"ContainerDied","Data":"1e211c1ce167cfd6209c6493f32afb5569adab4393472ba8db00e234d2d7f1cb"}
Oct 30 21:59:00.078439 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.078412    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:59:00.081675 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.081652    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:59:00.095980 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.095948    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"]
Oct 30 21:59:00.096542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096455    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1f4410dd-c22c-4cc4-9632-80166b19c500" containerName="revoke-keys"
Oct 30 21:59:00.096542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096469    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f4410dd-c22c-4cc4-9632-80166b19c500" containerName="revoke-keys"
Oct 30 21:59:00.096542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096530    4882 cpu_manager.go:414] "containerMap: removing container" podUID="750c3524-e7fe-452d-9f22-b0059092ee36" containerName="revoke-keys"
Oct 30 21:59:00.096542 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096537    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="750c3524-e7fe-452d-9f22-b0059092ee36" containerName="revoke-keys"
Oct 30 21:59:00.096781 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096648    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="750c3524-e7fe-452d-9f22-b0059092ee36" containerName="revoke-keys"
Oct 30 21:59:00.096781 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.096659    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f4410dd-c22c-4cc4-9632-80166b19c500" containerName="revoke-keys"
Oct 30 21:59:00.100104 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.100086    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.156283 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.156206    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"]
Oct 30 21:59:00.175716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.175685    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\") pod \"750c3524-e7fe-452d-9f22-b0059092ee36\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") "
Oct 30 21:59:00.175891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.175732    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\") pod \"1f4410dd-c22c-4cc4-9632-80166b19c500\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") "
Oct 30 21:59:00.175891 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.175873    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\") pod \"1f4410dd-c22c-4cc4-9632-80166b19c500\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") "
Oct 30 21:59:00.176020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.175957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\" (UniqueName: \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\") pod \"1f4410dd-c22c-4cc4-9632-80166b19c500\" (UID: \"1f4410dd-c22c-4cc4-9632-80166b19c500\") "
Oct 30 21:59:00.176020 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176014    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\") pod \"750c3524-e7fe-452d-9f22-b0059092ee36\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") "
Oct 30 21:59:00.176123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176081    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca" pod "1f4410dd-c22c-4cc4-9632-80166b19c500" (UID: "1f4410dd-c22c-4cc4-9632-80166b19c500"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:00.176123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176097    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\" (UniqueName: \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\") pod \"750c3524-e7fe-452d-9f22-b0059092ee36\" (UID: \"750c3524-e7fe-452d-9f22-b0059092ee36\") "
Oct 30 21:59:00.176221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176120    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca" pod "750c3524-e7fe-452d-9f22-b0059092ee36" (UID: "750c3524-e7fe-452d-9f22-b0059092ee36"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:00.176371 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176298    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.176463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176384    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42c98\" (UniqueName: \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.176659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176614    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.176815 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176787    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.176881 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.176817    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.178744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.178707    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca" pod "750c3524-e7fe-452d-9f22-b0059092ee36" (UID: "750c3524-e7fe-452d-9f22-b0059092ee36"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:00.178863 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.178782    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n" pod "1f4410dd-c22c-4cc4-9632-80166b19c500" (UID: "1f4410dd-c22c-4cc4-9632-80166b19c500"). InnerVolumeSpecName "kube-api-access-x5p8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:00.178863 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.178792    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca" pod "1f4410dd-c22c-4cc4-9632-80166b19c500" (UID: "1f4410dd-c22c-4cc4-9632-80166b19c500"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:00.178863 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.178811    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt" pod "750c3524-e7fe-452d-9f22-b0059092ee36" (UID: "750c3524-e7fe-452d-9f22-b0059092ee36"). InnerVolumeSpecName "kube-api-access-dqntt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:00.277575 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277538    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.277765 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277578    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42c98\" (UniqueName: \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.277807 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277777    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.277909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277894    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1f4410dd-c22c-4cc4-9632-80166b19c500-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.277947 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277915    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5p8n\" (UniqueName: \"kubernetes.io/projected/1f4410dd-c22c-4cc4-9632-80166b19c500-kube-api-access-x5p8n\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.277947 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277929    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/750c3524-e7fe-452d-9f22-b0059092ee36-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.278013 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.277952    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqntt\" (UniqueName: \"kubernetes.io/projected/750c3524-e7fe-452d-9f22-b0059092ee36-kube-api-access-dqntt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:00.278320 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.278302    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.280187 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.280164    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.340101 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.340058    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42c98\" (UniqueName: \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\") pod \"maas-api-revoke-sub-df0bbd04-94mt2\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.413502 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.413413    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:00.795806 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:00.795779    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa58f37_bfe7_4889_8666_f9c0a89086c6.slice/crio-6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46 WatchSource:0}: Error finding container 6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46: Status 404 returned error can't find the container with id 6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46
Oct 30 21:59:00.799889 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.799862    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"]
Oct 30 21:59:00.906820 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.906791    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"
Oct 30 21:59:00.906820 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.906814    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4" event={"ID":"750c3524-e7fe-452d-9f22-b0059092ee36","Type":"ContainerDied","Data":"a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b"}
Oct 30 21:59:00.907029 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.906869    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a0e1311e32825aa01da0501f50e6b7e157a6f77d2653eefbd875dc1e1b618c9b"
Oct 30 21:59:00.908201 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.908169    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"
Oct 30 21:59:00.908358 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.908197    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd" event={"ID":"1f4410dd-c22c-4cc4-9632-80166b19c500","Type":"ContainerDied","Data":"8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b"}
Oct 30 21:59:00.908358 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.908236    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8380d63096e3a58887b097ff8cdac5e17c51dcbebfc74f18296712cbae726f7b"
Oct 30 21:59:00.909639 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:00.909619    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2" event={"ID":"eaa58f37-bfe7-4889-8666-f9c0a89086c6","Type":"ContainerStarted","Data":"6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46"}
Oct 30 21:59:01.917524 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:01.917438    4882 generic.go:356] "Generic (PLEG): container finished" podID="eaa58f37-bfe7-4889-8666-f9c0a89086c6" containerID="63b8fbdc847e60888754841394cb04e3e634d0172662c46fc0428f20bd816a3a" exitCode=0
Oct 30 21:59:01.917958 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:01.917533    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2" event={"ID":"eaa58f37-bfe7-4889-8666-f9c0a89086c6","Type":"ContainerDied","Data":"63b8fbdc847e60888754841394cb04e3e634d0172662c46fc0428f20bd816a3a"}
Oct 30 21:59:03.061213 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.061183    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"]
Oct 30 21:59:03.079431 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.079402    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:03.110782 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.110746    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5c9e9eb3-k7vc4"]
Oct 30 21:59:03.112182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.112160    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\") pod \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") "
Oct 30 21:59:03.112336 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.112319    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\" (UniqueName: \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\") pod \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") "
Oct 30 21:59:03.112418 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.112400    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\") pod \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\" (UID: \"eaa58f37-bfe7-4889-8666-f9c0a89086c6\") "
Oct 30 21:59:03.112607 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.112570    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca" pod "eaa58f37-bfe7-4889-8666-f9c0a89086c6" (UID: "eaa58f37-bfe7-4889-8666-f9c0a89086c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:03.112861 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.112844    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.114477 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.114451    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca" pod "eaa58f37-bfe7-4889-8666-f9c0a89086c6" (UID: "eaa58f37-bfe7-4889-8666-f9c0a89086c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:03.114579 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.114536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98" pod "eaa58f37-bfe7-4889-8666-f9c0a89086c6" (UID: "eaa58f37-bfe7-4889-8666-f9c0a89086c6"). InnerVolumeSpecName "kube-api-access-42c98". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:03.214868 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.214786    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42c98\" (UniqueName: \"kubernetes.io/projected/eaa58f37-bfe7-4889-8666-f9c0a89086c6-kube-api-access-42c98\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.214868 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.214828    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaa58f37-bfe7-4889-8666-f9c0a89086c6-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:03.932665 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.932628    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"
Oct 30 21:59:03.932886 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.932627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2" event={"ID":"eaa58f37-bfe7-4889-8666-f9c0a89086c6","Type":"ContainerDied","Data":"6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46"}
Oct 30 21:59:03.932886 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:03.932745    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e18f1310a6daceea04de49ea41c20c37fb8d258786969b665ac3a8162a6cf46"
Oct 30 21:59:04.947969 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:04.947930    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="644a7b86-0693-46b5-bfa5-6f370b334013" path="/var/lib/kubelet/pods/644a7b86-0693-46b5-bfa5-6f370b334013/volumes"
Oct 30 21:59:07.228368 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.228278    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"]
Oct 30 21:59:07.229033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.229011    4882 cpu_manager.go:414] "containerMap: removing container" podUID="eaa58f37-bfe7-4889-8666-f9c0a89086c6" containerName="revoke-keys"
Oct 30 21:59:07.229127 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.229037    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaa58f37-bfe7-4889-8666-f9c0a89086c6" containerName="revoke-keys"
Oct 30 21:59:07.229243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.229228    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaa58f37-bfe7-4889-8666-f9c0a89086c6" containerName="revoke-keys"
Oct 30 21:59:07.233927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.233904    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.338796 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.338764    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:59:07.338796 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.338764    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:07.342493 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.342464    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"]
Oct 30 21:59:07.393757 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.393718    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5p97\" (UniqueName: \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.393996 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.393975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.394142 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.394119    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.495453 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.495368    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.495453 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.495434    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.495677 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.495481    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5p97\" (UniqueName: \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.496089 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.496057    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.497909 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.497882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.571078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.571043    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5p97\" (UniqueName: \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\") pod \"maas-api-revoke-sub-1fe0330d-dnzs2\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:07.847607 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:07.847543    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:08.225627 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:08.225562    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod523191c8_0779_48a7_b681_6e7daa54b12c.slice/crio-57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b WatchSource:0}: Error finding container 57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b: Status 404 returned error can't find the container with id 57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b
Oct 30 21:59:08.228800 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:08.228772    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"]
Oct 30 21:59:08.970692 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:08.970650    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2" event={"ID":"523191c8-0779-48a7-b681-6e7daa54b12c","Type":"ContainerStarted","Data":"2d23271b266dbd11fbc30e4917f267af9c0336a6a1a04bc763c01ab37de7674b"}
Oct 30 21:59:08.970880 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:08.970709    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2" event={"ID":"523191c8-0779-48a7-b681-6e7daa54b12c","Type":"ContainerStarted","Data":"57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b"}
Oct 30 21:59:09.089675 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:09.089607    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2" podStartSLOduration=2.089578007 podStartE2EDuration="2.089578007s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:09.088560225 +0000 UTC m=+1272.261884653" watchObservedRunningTime="2026-10-30 21:59:09.089578007 +0000 UTC m=+1272.262902464"
Oct 30 21:59:09.978216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:09.978183    4882 generic.go:356] "Generic (PLEG): container finished" podID="523191c8-0779-48a7-b681-6e7daa54b12c" containerID="2d23271b266dbd11fbc30e4917f267af9c0336a6a1a04bc763c01ab37de7674b" exitCode=0
Oct 30 21:59:09.978639 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:09.978268    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2" event={"ID":"523191c8-0779-48a7-b681-6e7daa54b12c","Type":"ContainerDied","Data":"2d23271b266dbd11fbc30e4917f267af9c0336a6a1a04bc763c01ab37de7674b"}
Oct 30 21:59:11.127869 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.127840    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:11.240936 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.240852    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\") pod \"523191c8-0779-48a7-b681-6e7daa54b12c\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") "
Oct 30 21:59:11.241103 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.240958    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\" (UniqueName: \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\") pod \"523191c8-0779-48a7-b681-6e7daa54b12c\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") "
Oct 30 21:59:11.241103 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.241018    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\") pod \"523191c8-0779-48a7-b681-6e7daa54b12c\" (UID: \"523191c8-0779-48a7-b681-6e7daa54b12c\") "
Oct 30 21:59:11.241221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.241198    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca" pod "523191c8-0779-48a7-b681-6e7daa54b12c" (UID: "523191c8-0779-48a7-b681-6e7daa54b12c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:11.241360 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.241345    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.243250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.243217    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca" pod "523191c8-0779-48a7-b681-6e7daa54b12c" (UID: "523191c8-0779-48a7-b681-6e7daa54b12c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:11.243358 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.243256    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97" pod "523191c8-0779-48a7-b681-6e7daa54b12c" (UID: "523191c8-0779-48a7-b681-6e7daa54b12c"). InnerVolumeSpecName "kube-api-access-n5p97". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:11.342413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.342369    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5p97\" (UniqueName: \"kubernetes.io/projected/523191c8-0779-48a7-b681-6e7daa54b12c-kube-api-access-n5p97\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.342413 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.342406    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/523191c8-0779-48a7-b681-6e7daa54b12c-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:11.748054 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.748019    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"]
Oct 30 21:59:11.748536 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.748523    4882 cpu_manager.go:414] "containerMap: removing container" podUID="523191c8-0779-48a7-b681-6e7daa54b12c" containerName="revoke-keys"
Oct 30 21:59:11.748631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.748538    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="523191c8-0779-48a7-b681-6e7daa54b12c" containerName="revoke-keys"
Oct 30 21:59:11.748674 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.748652    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="523191c8-0779-48a7-b681-6e7daa54b12c" containerName="revoke-keys"
Oct 30 21:59:11.752133 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.752110    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.808699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.808664    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"]
Oct 30 21:59:11.848183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.847785    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.848183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.847949    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.848183 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.848059    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btggf\" (UniqueName: \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.949380 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.949344    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.949569 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.949427    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btggf\" (UniqueName: \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.949569 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.949464    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.950073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.950052    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.951912 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.951890    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:11.992088 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.992057    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"
Oct 30 21:59:11.992233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.992098    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2" event={"ID":"523191c8-0779-48a7-b681-6e7daa54b12c","Type":"ContainerDied","Data":"57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b"}
Oct 30 21:59:11.992233 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:11.992151    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="57a6cc73ac43e02d0b13412ddd3a5bb274845d0c95d85c98447d77e11abfe05b"
Oct 30 21:59:12.011882 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:12.011857    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btggf\" (UniqueName: \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\") pod \"maas-api-revoke-sub-c48c0b73-fjn2k\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:12.066344 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:12.066308    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:12.470552 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:12.470516    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ea5714_7f7e_46d2_9ebf_3d8994ea7a5f.slice/crio-3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33 WatchSource:0}: Error finding container 3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33: Status 404 returned error can't find the container with id 3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33
Oct 30 21:59:12.473977 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:12.473956    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"]
Oct 30 21:59:13.000340 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:13.000261    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k" event={"ID":"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f","Type":"ContainerStarted","Data":"8584eed5c2e1b305d800beb49a6e29b36b3b409d8a198d51e4970dcac32598f2"}
Oct 30 21:59:13.000340 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:13.000311    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k" event={"ID":"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f","Type":"ContainerStarted","Data":"3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33"}
Oct 30 21:59:13.128646 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:13.128571    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k" podStartSLOduration=2.128553584 podStartE2EDuration="2.128553584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:13.125581742 +0000 UTC m=+1276.298906169" watchObservedRunningTime="2026-10-30 21:59:13.128553584 +0000 UTC m=+1276.301878010"
Oct 30 21:59:14.008097 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:14.008064    4882 generic.go:356] "Generic (PLEG): container finished" podID="39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" containerID="8584eed5c2e1b305d800beb49a6e29b36b3b409d8a198d51e4970dcac32598f2" exitCode=0
Oct 30 21:59:14.008580 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:14.008151    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k" event={"ID":"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f","Type":"ContainerDied","Data":"8584eed5c2e1b305d800beb49a6e29b36b3b409d8a198d51e4970dcac32598f2"}
Oct 30 21:59:15.153686 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.153657    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:15.285716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.285637    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\" (UniqueName: \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\") pod \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") "
Oct 30 21:59:15.285716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.285701    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\") pod \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") "
Oct 30 21:59:15.285927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.285813    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\") pod \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\" (UID: \"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f\") "
Oct 30 21:59:15.286045 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.286020    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca" pod "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" (UID: "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:15.286338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.286311    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:15.287885 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.287855    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca" pod "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" (UID: "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:15.288000 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.287979    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf" pod "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" (UID: "39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f"). InnerVolumeSpecName "kube-api-access-btggf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:15.387906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.387867    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btggf\" (UniqueName: \"kubernetes.io/projected/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-kube-api-access-btggf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:15.387906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:15.387902    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:16.024676 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.024631    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k" event={"ID":"39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f","Type":"ContainerDied","Data":"3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33"}
Oct 30 21:59:16.024937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.024911    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bfe23822a071a3d1d07748de875f227a941b7056e48dfff097c6e8fe15a6c33"
Oct 30 21:59:16.024937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.024753    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"
Oct 30 21:59:16.066338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.066305    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"]
Oct 30 21:59:16.116616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.116572    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-55162fd0-hgpsl"]
Oct 30 21:59:16.943732 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:16.943697    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0b77d4cb-5b07-448d-952a-945ca8ebf299" path="/var/lib/kubelet/pods/0b77d4cb-5b07-448d-952a-945ca8ebf299/volumes"
Oct 30 21:59:18.501033 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.500999    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"]
Oct 30 21:59:18.501505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.501491    4882 cpu_manager.go:414] "containerMap: removing container" podUID="39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" containerName="revoke-keys"
Oct 30 21:59:18.501550 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.501506    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" containerName="revoke-keys"
Oct 30 21:59:18.501632 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.501621    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" containerName="revoke-keys"
Oct 30 21:59:18.505312 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.505295    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.523342 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.523311    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnqb\" (UniqueName: \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.523458 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.523352    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.523516 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.523457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.559764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.559729    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"]
Oct 30 21:59:18.605780 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.605742    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:18.605937 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.605799    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:59:18.624453 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.624423    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.624662 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.624538    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnqb\" (UniqueName: \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.624662 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.624563    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.625087 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.625064    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.626914 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.626891    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.686731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.686692    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnqb\" (UniqueName: \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\") pod \"maas-api-revoke-sub-9f11b7b5-652d4\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:18.819952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:18.819866    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:19.056530 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:19.056492    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"]
Oct 30 21:59:19.064749 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:19.064711    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bbe53b4d-gpz9h"]
Oct 30 21:59:19.193569 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:19.193534    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a5fc07_1c1d_4e3d_8b90_d98acba23cab.slice/crio-d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83 WatchSource:0}: Error finding container d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83: Status 404 returned error can't find the container with id d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83
Oct 30 21:59:19.197050 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:19.197025    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"]
Oct 30 21:59:20.052704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:20.052671    4882 generic.go:356] "Generic (PLEG): container finished" podID="73a5fc07-1c1d-4e3d-8b90-d98acba23cab" containerID="e934ff984e9bb4f4453fbb088514eb72cc677c643ed869ba0a9cf306f4cfa1ed" exitCode=0
Oct 30 21:59:20.053153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:20.052722    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4" event={"ID":"73a5fc07-1c1d-4e3d-8b90-d98acba23cab","Type":"ContainerDied","Data":"e934ff984e9bb4f4453fbb088514eb72cc677c643ed869ba0a9cf306f4cfa1ed"}
Oct 30 21:59:20.053153 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:20.052786    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4" event={"ID":"73a5fc07-1c1d-4e3d-8b90-d98acba23cab","Type":"ContainerStarted","Data":"d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83"}
Oct 30 21:59:20.808450 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:59:20.808409    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:59:20.943865 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:20.943830    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="325b522f-e276-4f02-9cb2-ea34bc9869b1" path="/var/lib/kubelet/pods/325b522f-e276-4f02-9cb2-ea34bc9869b1/volumes"
Oct 30 21:59:21.208021 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.207992    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:21.255741 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.255702    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\" (UniqueName: \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\") pod \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") "
Oct 30 21:59:21.255938 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.255829    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\") pod \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") "
Oct 30 21:59:21.255938 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.255880    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\") pod \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\" (UID: \"73a5fc07-1c1d-4e3d-8b90-d98acba23cab\") "
Oct 30 21:59:21.256282 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.256246    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca" pod "73a5fc07-1c1d-4e3d-8b90-d98acba23cab" (UID: "73a5fc07-1c1d-4e3d-8b90-d98acba23cab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:21.256379 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.256367    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:21.257810 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.257788    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb" pod "73a5fc07-1c1d-4e3d-8b90-d98acba23cab" (UID: "73a5fc07-1c1d-4e3d-8b90-d98acba23cab"). InnerVolumeSpecName "kube-api-access-znnqb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:21.257916 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.257865    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca" pod "73a5fc07-1c1d-4e3d-8b90-d98acba23cab" (UID: "73a5fc07-1c1d-4e3d-8b90-d98acba23cab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:21.357146 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.357114    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnqb\" (UniqueName: \"kubernetes.io/projected/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-kube-api-access-znnqb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:21.357146 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:21.357142    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73a5fc07-1c1d-4e3d-8b90-d98acba23cab-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:22.067232 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:22.067199    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4" event={"ID":"73a5fc07-1c1d-4e3d-8b90-d98acba23cab","Type":"ContainerDied","Data":"d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83"}
Oct 30 21:59:22.067388 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:22.067244    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5e35227076e2ec8e86c7e08f0b735a56b69c97caccb5c8509dd441b6aed0d83"
Oct 30 21:59:22.067388 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:22.067214    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"
Oct 30 21:59:24.310073 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.310037    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 21:59:24.310887 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.310866    4882 cpu_manager.go:414] "containerMap: removing container" podUID="73a5fc07-1c1d-4e3d-8b90-d98acba23cab" containerName="revoke-keys"
Oct 30 21:59:24.310887 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.310887    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73a5fc07-1c1d-4e3d-8b90-d98acba23cab" containerName="revoke-keys"
Oct 30 21:59:24.311007 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.310989    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="73a5fc07-1c1d-4e3d-8b90-d98acba23cab" containerName="revoke-keys"
Oct 30 21:59:24.314560 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.314537    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390484 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390447    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390678 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390521    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390678 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390550    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390783 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390754    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390837 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390799    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390906 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390881    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.390984 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390971    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gsv5k\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.391019 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.390994    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.391085 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.391069    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.402939 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.402907    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 21:59:24.492505 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492466    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492532    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492618    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492647    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492740    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492766    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492790    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492861    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gsv5k\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.492922 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.492889    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.493134 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.493058    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.493250 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.493221    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.493362 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.493264    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.493362 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.493331    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.493438 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.493359    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.495407 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.495380    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.495652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.495635    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.554339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.554305    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gsv5k\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.554474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.554381    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:24.629573 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:24.629537    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:25.062300 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:25.062245    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0737f847_c26f_4fba_8790_cb5b7875e88f.slice/crio-005b8306cc1aa4a27340827dba6c689bf5a19c768a7d5b0564b4953d9dad495f WatchSource:0}: Error finding container 005b8306cc1aa4a27340827dba6c689bf5a19c768a7d5b0564b4953d9dad495f: Status 404 returned error can't find the container with id 005b8306cc1aa4a27340827dba6c689bf5a19c768a7d5b0564b4953d9dad495f
Oct 30 21:59:25.064781 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.064748    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:25.064892 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.064832    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:25.064892 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.064881    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:25.065564 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.065540    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 21:59:25.090926 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.090894    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" event={"ID":"0737f847-c26f-4fba-8790-cb5b7875e88f","Type":"ContainerStarted","Data":"005b8306cc1aa4a27340827dba6c689bf5a19c768a7d5b0564b4953d9dad495f"}
Oct 30 21:59:25.450652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.450612    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 21:59:25.454884 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.454864    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.506745 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.506713    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.506896 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.506852    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.506896 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.506890    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.506982    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.507038    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt2wh\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507095 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.507067    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507163 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.507116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.507179    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.507243 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.507217    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.532768 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.532732    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.608886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.608991    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609035    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609073    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609125    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt2wh\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609153    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609176    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.609273    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.610310    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.610539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.610745    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.611030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.612394 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.612073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.615759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.615032    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.615759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.615621    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.693367 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.693330    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt2wh\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.693598 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.693558    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:25.770673 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:25.770581    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:26.098749 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.098708    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" event={"ID":"0737f847-c26f-4fba-8790-cb5b7875e88f","Type":"ContainerStarted","Data":"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"}
Oct 30 21:59:26.236686 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.224768    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podStartSLOduration=2.224750085 podStartE2EDuration="2.224750085s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:26.223781154 +0000 UTC m=+1289.397105584" watchObservedRunningTime="2026-10-30 21:59:26.224750085 +0000 UTC m=+1289.398074514"
Oct 30 21:59:26.257678 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:26.257645    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b41a63c_ea14_44f0_9ade_811b708624ed.slice/crio-c5321ab9ce21fea42b123852f73f5ddf69a7ea4a5b4862f1241386b656d854c1 WatchSource:0}: Error finding container c5321ab9ce21fea42b123852f73f5ddf69a7ea4a5b4862f1241386b656d854c1: Status 404 returned error can't find the container with id c5321ab9ce21fea42b123852f73f5ddf69a7ea4a5b4862f1241386b656d854c1
Oct 30 21:59:26.260184 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.260148    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:26.260305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.260242    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:26.260305 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.260287    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:26.277886 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.277850    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 21:59:26.646670 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.645983    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:26.655811 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.655772    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:26.655979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:26.655847    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:27.153612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.153106    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" event={"ID":"8b41a63c-ea14-44f0-9ade-811b708624ed","Type":"ContainerStarted","Data":"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"}
Oct 30 21:59:27.153612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.153168    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" event={"ID":"8b41a63c-ea14-44f0-9ade-811b708624ed","Type":"ContainerStarted","Data":"c5321ab9ce21fea42b123852f73f5ddf69a7ea4a5b4862f1241386b656d854c1"}
Oct 30 21:59:27.304078 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.304023    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podStartSLOduration=2.30400816 podStartE2EDuration="2.30400816s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:27.300929466 +0000 UTC m=+1290.474253893" watchObservedRunningTime="2026-10-30 21:59:27.30400816 +0000 UTC m=+1290.477332587"
Oct 30 21:59:27.633623 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.633033    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:27.633623 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.633096    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:27.772613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.771981    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:27.774090 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.774052    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:27.774221 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:27.774132    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:28.351631 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:28.351562    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 21:59:28.644667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:28.635046    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:28.644667 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:28.635114    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:28.774011 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:28.773965    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:28.774572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:28.774537    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:29.289625 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.286400    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 21:59:29.300171 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.300141    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.348612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.339385    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 21:59:29.371736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.357022    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.371736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.357183    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"]
Oct 30 21:59:29.388608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.376385    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.405822 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.402436    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418675    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418762    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-djpbh\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418800    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418827    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418889    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418919    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418955    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.418998    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.419445 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.419074    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.438752 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.438721    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:59:29.438990 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.438971    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:29.492838 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.475679    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613280    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613353    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613412    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613445    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613552    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-djpbh\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613581    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktx9t\" (UniqueName: \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613636    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613663    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613704    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613755    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613782    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xw6qd\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613814    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613853    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613914    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.613955    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614023    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614062    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614095    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614153    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614198    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.614784    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.615422    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.660672 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.649378    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.699614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.682982    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.699809 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.699745    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.733187    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.733255    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736653    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736736    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736777    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736863    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736902    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.736957    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.753652    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-djpbh\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.782247    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.785000    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.785052    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.790174    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.800832 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.790419    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.806896    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807023    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807307    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807364    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807469    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktx9t\" (UniqueName: \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807527    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.807577    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xw6qd\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.812613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.811181    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.815151 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.815070    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:29.815999 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.815946    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.835648 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.833283    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:29.836641 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.836532    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:29.902609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.872920    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.966078    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.966776    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.954935    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"]
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.979447    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.985993    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktx9t\" (UniqueName: \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\") pod \"maas-api-revoke-sub-1591f433-b8s7z\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:30.003661 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:29.992030    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xw6qd\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:30.145611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.105689    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:30.182613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.171102    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:30.391616 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.389758    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:30.633613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.632061    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:30.633613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.632145    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:30.672623 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.658122    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:30.784613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.781546    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:30.784613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.781639    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:30.790647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.787934    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:30.895343 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.895292    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:30.966120 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:30.966072    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.062784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.061838    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.062784 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.061959    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": context deadline exceeded"
Oct 30 21:59:31.269988 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.266170    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.584673 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.571852    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.682612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.679759    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.682612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.680297    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:59:31.682612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.682167    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:31.682612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.682212    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:31.797770 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.781977    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:31.797770 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.782038    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:31.897277 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.896038    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:31.897277 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.896485    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:59:31.921150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:31.921121    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:59:32.168014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.164441    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:32.168014 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.164789    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:59:32.175613 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:32.171714    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod673ea3b3_f9ae_4e71_83b5_eddeaf89299e.slice/crio-370d3a64106e56d68e5b37719e9402fd4470caba7d01a23232115b5118063427 WatchSource:0}: Error finding container 370d3a64106e56d68e5b37719e9402fd4470caba7d01a23232115b5118063427: Status 404 returned error can't find the container with id 370d3a64106e56d68e5b37719e9402fd4470caba7d01a23232115b5118063427
Oct 30 21:59:32.207114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.175837    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 21:59:32.207114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.184180    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:32.207114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.184263    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:32.207114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.184308    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:32.207114 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.206470    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"]
Oct 30 21:59:32.216892 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.213932    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5402be1-vbjkw"]
Oct 30 21:59:32.346175 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.341718    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:59:32.346175 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.344910    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:32.630930 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.630155    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:32.630930 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.630211    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:32.647454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.647425    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"]
Oct 30 21:59:32.783276 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.783183    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:32.783561 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:32.783514    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:33.051032 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.050960    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dddbe74e-726c-4378-8854-b37007689716" path="/var/lib/kubelet/pods/dddbe74e-726c-4378-8854-b37007689716/volumes"
Oct 30 21:59:33.052140 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.052055    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" event={"ID":"673ea3b3-f9ae-4e71-83b5-eddeaf89299e","Type":"ContainerStarted","Data":"370d3a64106e56d68e5b37719e9402fd4470caba7d01a23232115b5118063427"}
Oct 30 21:59:33.052330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.052313    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 21:59:33.052454 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.052438    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z" event={"ID":"4d6889ad-f9db-4c84-929d-d7aec3e005f6","Type":"ContainerStarted","Data":"fc2926f499e61da55ab96702d7b9c4b17e3cc47f98e771a14504d63dacc001c7"}
Oct 30 21:59:33.241640 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.239690    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:33.260716 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.260298    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:33.504609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.459763    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:33.504609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.459868    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:33.504609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.459918    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:33.504609 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.476683    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:33.907770 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.907731    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:33.997167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.952010    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:33.997167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.913490    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:33.997167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.952615    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:33.997167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.605250    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:33.997167 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:33.952711    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.059688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.058090    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.186958 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.186918    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" event={"ID":"673ea3b3-f9ae-4e71-83b5-eddeaf89299e","Type":"ContainerStarted","Data":"35bff7387427211f285ae61853c4702d4cd036d61a832a4bfd99003407db583a"}
Oct 30 21:59:34.194638 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.194579    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" event={"ID":"747e0db1-77d5-45bc-9a8f-c7f8154fd74a","Type":"ContainerStarted","Data":"cb128e91db541f759b352fa20a0b137658f78d51f735cc88902c6ef8f632de27"}
Oct 30 21:59:34.355786 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.355729    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.356314 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.356259    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.356678 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.356653    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/health\": context deadline exceeded"
Oct 30 21:59:34.357705 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.357661    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.374613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.369086    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.374613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.369517    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:59:34.517655 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.510513    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podStartSLOduration=5.510492221 podStartE2EDuration="5.510492221s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:34.359795867 +0000 UTC m=+1297.533120298" watchObservedRunningTime="2026-10-30 21:59:34.510492221 +0000 UTC m=+1297.683816649"
Oct 30 21:59:34.675222 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.675130    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused" start-of-body=
Oct 30 21:59:34.675478 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.675446    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": dial tcp 10.132.0.85:15021: connect: connection refused"
Oct 30 21:59:34.678742 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.518733    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:34.706195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.706152    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:59:34.772708 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.772645    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:34.772950 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.772923    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:34.789657 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.789629    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:59:34.821685 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.821644    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:59:34.821896 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.821728    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:59:35.001699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.998916    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:35.001699 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:34.998976    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:35.064979 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.062827    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:35.320300 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.320200    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d6889ad-f9db-4c84-929d-d7aec3e005f6" containerID="3e532c2350ccb9b2663458890d2f53de8260747a0dc96ae04891a455e7338e6c" exitCode=0
Oct 30 21:59:35.320662 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.320634    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z" event={"ID":"4d6889ad-f9db-4c84-929d-d7aec3e005f6","Type":"ContainerDied","Data":"3e532c2350ccb9b2663458890d2f53de8260747a0dc96ae04891a455e7338e6c"}
Oct 30 21:59:35.360731 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.360682    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" event={"ID":"747e0db1-77d5-45bc-9a8f-c7f8154fd74a","Type":"ContainerStarted","Data":"c318244639b61a5a370e76ebc707208f98ad85ed3a5db3ec77d8159f197ee2da"}
Oct 30 21:59:35.505717 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.505661    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podStartSLOduration=6.505645146 podStartE2EDuration="6.505645146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:35.501752204 +0000 UTC m=+1298.675076649" watchObservedRunningTime="2026-10-30 21:59:35.505645146 +0000 UTC m=+1298.678969573"
Oct 30 21:59:35.772474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.772035    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused" start-of-body=
Oct 30 21:59:35.772474 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.772098    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": dial tcp 10.132.0.86:15021: connect: connection refused"
Oct 30 21:59:36.000634 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.997501    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:36.000634 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:35.997564    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:36.107902 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:36.107841    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:36.109338 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:36.109307    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:36.109496 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:36.109372    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:36.685821 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:36.683767    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:36.685821 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:36.683835    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:37.019691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.002072    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:37.019691 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.002137    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:37.120818 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.112028    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:37.120818 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.112092    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:37.197652 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.182746    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"]
Oct 30 21:59:37.212901 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.212872    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.294959 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.269559    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"]
Oct 30 21:59:37.382611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.366084    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvsg\" (UniqueName: \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.382611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.369661    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.382611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.375236    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.473353 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.473265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z" event={"ID":"4d6889ad-f9db-4c84-929d-d7aec3e005f6","Type":"ContainerDied","Data":"fc2926f499e61da55ab96702d7b9c4b17e3cc47f98e771a14504d63dacc001c7"}
Oct 30 21:59:37.473353 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.473323    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc2926f499e61da55ab96702d7b9c4b17e3cc47f98e771a14504d63dacc001c7"
Oct 30 21:59:37.478443 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.478375    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.484608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.479411    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvsg\" (UniqueName: \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.484894 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.484866    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.489617 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.485572    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.493833 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.493403    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:37.502041 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.501211    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.561973 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.561832    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:37.583574 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.582384    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvsg\" (UniqueName: \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\") pod \"maas-api-revoke-sub-f829486c-s6dq4\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.583844 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.583821    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:37.591075 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.591039    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\") pod \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") "
Oct 30 21:59:37.592457 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.592431    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\" (UniqueName: \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\") pod \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") "
Oct 30 21:59:37.592721 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.592699    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\") pod \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\" (UID: \"4d6889ad-f9db-4c84-929d-d7aec3e005f6\") "
Oct 30 21:59:37.594159 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.592281    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca" pod "4d6889ad-f9db-4c84-929d-d7aec3e005f6" (UID: "4d6889ad-f9db-4c84-929d-d7aec3e005f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:37.609182 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.608997    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 21:59:37.611042 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.611007    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t" pod "4d6889ad-f9db-4c84-929d-d7aec3e005f6" (UID: "4d6889ad-f9db-4c84-929d-d7aec3e005f6"). InnerVolumeSpecName "kube-api-access-ktx9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:37.611570 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.611538    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca" pod "4d6889ad-f9db-4c84-929d-d7aec3e005f6" (UID: "4d6889ad-f9db-4c84-929d-d7aec3e005f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:37.696194 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.696148    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:37.696194 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.696182    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktx9t\" (UniqueName: \"kubernetes.io/projected/4d6889ad-f9db-4c84-929d-d7aec3e005f6-kube-api-access-ktx9t\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:37.696194 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.696200    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4d6889ad-f9db-4c84-929d-d7aec3e005f6-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:37.776300 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.776251    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:37.777418 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.777360    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:37.777681 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.777641    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:37.989613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.986411    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:37.989613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:37.986474    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:38.108927 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.108877    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:38.109088 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.108948    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:38.260613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.260454    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"]
Oct 30 21:59:38.497608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.494006    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4" event={"ID":"027a9be8-074d-459d-946f-93c060a0636f","Type":"ContainerStarted","Data":"26f0587a5bef87bf7d9edb1941a80ac8375be90ca82a6fc8ddca97cc93327414"}
Oct 30 21:59:38.497608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.494115    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"
Oct 30 21:59:38.825608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.824618    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:38.825608 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.824681    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:38.994103 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.994054    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:38.994294 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:38.994136    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:39.011965 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.011930    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:39.012241 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.012218    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:39.013147 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.013122    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 21:59:39.108076 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.108040    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:39.108258 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.108091    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:39.502623 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.502575    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4" event={"ID":"027a9be8-074d-459d-946f-93c060a0636f","Type":"ContainerStarted","Data":"a715f11a8adcba2e0d0abaca8c7d947911f7ba53371d3b84156b1d9f76bce690"}
Oct 30 21:59:39.648611 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.648478    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4" podStartSLOduration=2.648455828 podStartE2EDuration="2.648455828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:39.645008895 +0000 UTC m=+1302.818333323" watchObservedRunningTime="2026-10-30 21:59:39.648455828 +0000 UTC m=+1302.821780258"
Oct 30 21:59:39.985330 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.985294    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:39.985540 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:39.985464    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:40.108022 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.107990    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:40.108202 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.108055    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:40.122463 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.122429    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"]
Oct 30 21:59:40.180649 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.163414    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a121c921-2c4lv"]
Oct 30 21:59:40.511897 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.511861    4882 generic.go:356] "Generic (PLEG): container finished" podID="027a9be8-074d-459d-946f-93c060a0636f" containerID="a715f11a8adcba2e0d0abaca8c7d947911f7ba53371d3b84156b1d9f76bce690" exitCode=0
Oct 30 21:59:40.513119 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.513091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4" event={"ID":"027a9be8-074d-459d-946f-93c060a0636f","Type":"ContainerDied","Data":"a715f11a8adcba2e0d0abaca8c7d947911f7ba53371d3b84156b1d9f76bce690"}
Oct 30 21:59:40.585571 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.585530    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 21:59:40.586850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.586820    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d6889ad-f9db-4c84-929d-d7aec3e005f6" containerName="revoke-keys"
Oct 30 21:59:40.587005 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.586990    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d6889ad-f9db-4c84-929d-d7aec3e005f6" containerName="revoke-keys"
Oct 30 21:59:40.587268 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.587253    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d6889ad-f9db-4c84-929d-d7aec3e005f6" containerName="revoke-keys"
Oct 30 21:59:40.591704 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.591672    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.671553 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.671515    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 21:59:40.691531 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691494    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691776 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691661    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjcff\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691776 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691710    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691775    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691831    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691846    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.691900 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.692108 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.691966    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.692108 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.692068    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.793835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793747    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.793835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793782    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.793835 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793807    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793839    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793913    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.793946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794043    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjcff\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794072    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794128 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794539 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794442    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794539 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794491    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794659 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794610    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794758 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794727    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.794908 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.794885    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.796866 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.796841    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.796952 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.796870    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.856336 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.856300    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.857647 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.857618    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjcff\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.905775 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.905739    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:40.965293 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.965253    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="41db0538-37d9-4355-a3cd-ccef4b7d2f50" path="/var/lib/kubelet/pods/41db0538-37d9-4355-a3cd-ccef4b7d2f50/volumes"
Oct 30 21:59:40.985079 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.985035    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused" start-of-body=
Oct 30 21:59:40.985281 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:40.985101    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.87:15021/healthz/ready\": dial tcp 10.132.0.87:15021: connect: connection refused"
Oct 30 21:59:41.108850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.108806    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:41.109064 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.108867    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:41.372427 ip-10-0-141-82 kubenswrapper[4882]: W1030 21:59:41.372389    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee5b1749_e7b8_4b38_bbb3_777b0a01bb1a.slice/crio-cbbc982b5f971568904fb6d92dbe49ec3c3595257e0885edff9c2ce18b8cc60c WatchSource:0}: Error finding container cbbc982b5f971568904fb6d92dbe49ec3c3595257e0885edff9c2ce18b8cc60c: Status 404 returned error can't find the container with id cbbc982b5f971568904fb6d92dbe49ec3c3595257e0885edff9c2ce18b8cc60c
Oct 30 21:59:41.373764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.373736    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 21:59:41.376112 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.376067    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:41.376195 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.376162    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:41.376283 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.376210    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 21:59:41.527824 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.527780    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" event={"ID":"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a","Type":"ContainerStarted","Data":"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"}
Oct 30 21:59:41.528335 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.527855    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" event={"ID":"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a","Type":"ContainerStarted","Data":"cbbc982b5f971568904fb6d92dbe49ec3c3595257e0885edff9c2ce18b8cc60c"}
Oct 30 21:59:41.658123 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.657990    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podStartSLOduration=1.6579680799999998 podStartE2EDuration="1.65796808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 21:59:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 21:59:41.652342137 +0000 UTC m=+1304.825666575" watchObservedRunningTime="2026-10-30 21:59:41.65796808 +0000 UTC m=+1304.831292511"
Oct 30 21:59:41.705736 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.705706    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:41.807912 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.807871    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\") pod \"027a9be8-074d-459d-946f-93c060a0636f\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") "
Oct 30 21:59:41.808102 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.808072    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\" (UniqueName: \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\") pod \"027a9be8-074d-459d-946f-93c060a0636f\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") "
Oct 30 21:59:41.808180 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.808122    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\") pod \"027a9be8-074d-459d-946f-93c060a0636f\" (UID: \"027a9be8-074d-459d-946f-93c060a0636f\") "
Oct 30 21:59:41.808764 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.808733    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca" pod "027a9be8-074d-459d-946f-93c060a0636f" (UID: "027a9be8-074d-459d-946f-93c060a0636f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:41.811056 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.811025    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg" pod "027a9be8-074d-459d-946f-93c060a0636f" (UID: "027a9be8-074d-459d-946f-93c060a0636f"). InnerVolumeSpecName "kube-api-access-8vvsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:41.812228 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.812198    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca" pod "027a9be8-074d-459d-946f-93c060a0636f" (UID: "027a9be8-074d-459d-946f-93c060a0636f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:41.906420 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.906383    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:41.910552 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.910465    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/027a9be8-074d-459d-946f-93c060a0636f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:41.910552 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.910510    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvsg\" (UniqueName: \"kubernetes.io/projected/027a9be8-074d-459d-946f-93c060a0636f-kube-api-access-8vvsg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:41.910552 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:41.910531    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/027a9be8-074d-459d-946f-93c060a0636f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:42.108614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.108475    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused" start-of-body=
Oct 30 21:59:42.108614 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.108535    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": dial tcp 10.132.0.88:15021: connect: connection refused"
Oct 30 21:59:42.203322 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.203242    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:42.203745 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.203718    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:42.204663 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.204641    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 21:59:42.570559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.570433    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"
Oct 30 21:59:42.570559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.570445    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4" event={"ID":"027a9be8-074d-459d-946f-93c060a0636f","Type":"ContainerDied","Data":"26f0587a5bef87bf7d9edb1941a80ac8375be90ca82a6fc8ddca97cc93327414"}
Oct 30 21:59:42.570559 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.570512    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26f0587a5bef87bf7d9edb1941a80ac8375be90ca82a6fc8ddca97cc93327414"
Oct 30 21:59:42.908004 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.907971    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:42.908186 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:42.908035    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:43.113216 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:43.113175    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:43.113621 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:43.113597    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:43.114341 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:43.114298    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 21:59:43.907292 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:43.907191    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:43.908864 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:43.908116    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:44.907197 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:44.907159    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:44.907378 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:44.907220    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:45.474916 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.474877    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"]
Oct 30 21:59:45.475498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.475481    4882 cpu_manager.go:414] "containerMap: removing container" podUID="027a9be8-074d-459d-946f-93c060a0636f" containerName="revoke-keys"
Oct 30 21:59:45.475498 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.475497    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="027a9be8-074d-459d-946f-93c060a0636f" containerName="revoke-keys"
Oct 30 21:59:45.475698 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.475616    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="027a9be8-074d-459d-946f-93c060a0636f" containerName="revoke-keys"
Oct 30 21:59:45.479242 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.479218    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.488642 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.488617    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.488767 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.488652    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.488904 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.488882    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p2bhp\" (UniqueName: \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.558759 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.558710    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"]
Oct 30 21:59:45.589745 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.589709    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p2bhp\" (UniqueName: \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.589967 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.589788    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.589967 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.589821    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.590734 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.590698    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.595352 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.595325    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 21:59:45.595489 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.595368    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 21:59:45.602512 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.602484    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.655139 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.655056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p2bhp\" (UniqueName: \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\") pod \"maas-api-revoke-sub-9808ceaf-m2mzx\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.792572 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.792489    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:45.906812 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.906777    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:45.906983 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:45.906850    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:46.191389 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:46.191358    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"]
Oct 30 21:59:46.603005 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:46.602972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx" event={"ID":"4229bcaa-d1d8-491c-a588-2d519ead5a08","Type":"ContainerStarted","Data":"af9dfbc464bfdefee3c5c278a1e971f6e25d47edfac8d28b247beebce742d754"}
Oct 30 21:59:46.906668 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:46.906632    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:46.906828 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:46.906687    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:47.614200 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:47.614162    4882 generic.go:356] "Generic (PLEG): container finished" podID="4229bcaa-d1d8-491c-a588-2d519ead5a08" containerID="5e5055192cfde1480be9fcb683715afdd8f9e03715d563c4ea6394e1bdc094b4" exitCode=0
Oct 30 21:59:47.614650 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:47.614257    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx" event={"ID":"4229bcaa-d1d8-491c-a588-2d519ead5a08","Type":"ContainerDied","Data":"5e5055192cfde1480be9fcb683715afdd8f9e03715d563c4ea6394e1bdc094b4"}
Oct 30 21:59:47.906817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:47.906737    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused" start-of-body=
Oct 30 21:59:47.906817 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:47.906804    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.91:15021/healthz/ready\": dial tcp 10.132.0.91:15021: connect: connection refused"
Oct 30 21:59:48.797664 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.797630    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:48.834011 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.833977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\") pod \"4229bcaa-d1d8-491c-a588-2d519ead5a08\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") "
Oct 30 21:59:48.834186 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.834173    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\") pod \"4229bcaa-d1d8-491c-a588-2d519ead5a08\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") "
Oct 30 21:59:48.834328 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.834295    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\" (UniqueName: \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\") pod \"4229bcaa-d1d8-491c-a588-2d519ead5a08\" (UID: \"4229bcaa-d1d8-491c-a588-2d519ead5a08\") "
Oct 30 21:59:48.834637 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.834562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca" pod "4229bcaa-d1d8-491c-a588-2d519ead5a08" (UID: "4229bcaa-d1d8-491c-a588-2d519ead5a08"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 21:59:48.834872 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.834850    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:48.836439 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.836415    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca" pod "4229bcaa-d1d8-491c-a588-2d519ead5a08" (UID: "4229bcaa-d1d8-491c-a588-2d519ead5a08"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 21:59:48.836790 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.836769    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp" pod "4229bcaa-d1d8-491c-a588-2d519ead5a08" (UID: "4229bcaa-d1d8-491c-a588-2d519ead5a08"). InnerVolumeSpecName "kube-api-access-p2bhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 21:59:48.910785 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.910704    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:48.911125 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.911097    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:48.911789 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.911767    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 21:59:48.947470 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.947425    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p2bhp\" (UniqueName: \"kubernetes.io/projected/4229bcaa-d1d8-491c-a588-2d519ead5a08-kube-api-access-p2bhp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:48.947470 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:48.947466    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4229bcaa-d1d8-491c-a588-2d519ead5a08-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 21:59:49.630264 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:49.630237    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"
Oct 30 21:59:49.630434 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:49.630269    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx" event={"ID":"4229bcaa-d1d8-491c-a588-2d519ead5a08","Type":"ContainerDied","Data":"af9dfbc464bfdefee3c5c278a1e971f6e25d47edfac8d28b247beebce742d754"}
Oct 30 21:59:49.630434 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:49.630319    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af9dfbc464bfdefee3c5c278a1e971f6e25d47edfac8d28b247beebce742d754"
Oct 30 21:59:50.807574 ip-10-0-141-82 kubenswrapper[4882]: E1030 21:59:50.807530    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 21:59:51.052994 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:51.052963    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"]
Oct 30 21:59:51.103155 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:51.103116    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a0e95e13-zhr7v"]
Oct 30 21:59:52.943150 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:52.943102    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52fd999b-d1d5-4248-914e-f4cf525b7886" path="/var/lib/kubelet/pods/52fd999b-d1d5-4248-914e-f4cf525b7886/volumes"
Oct 30 21:59:57.076747 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.076707    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"]
Oct 30 21:59:57.115567 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.115496    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03173270-v7gr2"]
Oct 30 21:59:57.179613 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.170657    4882 scope.go:122] "RemoveContainer" containerID="d561546720c96389768bacef32d50e8cabc9ad0fe74678f71304a145cd80e7ba"
Oct 30 21:59:57.796754 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.796694    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:57.826748 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.826693    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:57.923773 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.923714    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:57.934801 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:57.934745    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 21:59:58.067863 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.067738    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.297795 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.297701    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.359728 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.359678    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.361339 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.361303    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.367713 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.367671    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.778668 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.763099    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:58.778668 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.763164    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.43:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.827744 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.827689    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:58.893612 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:58.831941    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:59:59.353727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.346086    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": context deadline exceeded"
Oct 30 21:59:59.353727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.346487    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" probeResult="failure" output="Get \"https://10.132.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:59.353727 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.348839    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": context deadline exceeded"
Oct 30 21:59:59.460760 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.422692    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": context deadline exceeded"
Oct 30 21:59:59.462928 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.461536    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 21:59:59.462928 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.461930    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:59.472688 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.428441    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": context deadline exceeded"
Oct 30 21:59:59.499654 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.498165    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" probeResult="failure" output="Get \"https://10.132.0.64:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:59.499850 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.499770    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" probeResult="failure" output="Get \"https://10.132.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 21:59:59.597821 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.597716    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7241e241-b3d4-4efd-9848-5a9841dfc48e" path="/var/lib/kubelet/pods/7241e241-b3d4-4efd-9848-5a9841dfc48e/volumes"
Oct 30 21:59:59.600262 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.600235    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:59:59.600518 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.600501    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 21:59:59.600753 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.600736    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:59:59.600912 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.600898    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:59:59.601075 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.601062    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:59:59.601811 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.601793    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:59:59.602034 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.602021    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:59:59.602145 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.602135    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 21:59:59.602273 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.602263    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 21:59:59.745211 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.745126    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 21:59:59.802277 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.799941    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 21:59:59.910172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.908334    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 21:59:59.910172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.909143    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 21:59:59.910172 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.910015    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 21:59:59.911055 ip-10-0-141-82 kubenswrapper[4882]: I1030 21:59:59.911033    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 22:00:00.303071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.303029    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:00.303632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.303611    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4229bcaa-d1d8-491c-a588-2d519ead5a08" containerName="revoke-keys"
Oct 30 22:00:00.303779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.303760    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4229bcaa-d1d8-491c-a588-2d519ead5a08" containerName="revoke-keys"
Oct 30 22:00:00.303947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.303931    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4229bcaa-d1d8-491c-a588-2d519ead5a08" containerName="revoke-keys"
Oct 30 22:00:00.318936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.318905    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:00.323198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.319270    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.388733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.388704    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:00:00.398829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.398804    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.404499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.404472    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:00.439636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.431975    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.439942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.439910    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wbhkh\" (UniqueName: \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.440376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.440351    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:00.440683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.440666    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:00.444007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.440598    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.461809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.461782    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:00.462118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.462095    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.475104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.475042    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:00.482527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.482494    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:00.542414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.542370    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wbhkh\" (UniqueName: \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.543072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.543041    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.543562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.543538    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.543739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.543717    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hwq4\" (UniqueName: \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.543898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.543880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7spv\" (UniqueName: \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.544015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.543999    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d47th\" (UniqueName: \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.544193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.544176    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.544319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.544303    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.548143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.547412    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.637880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.637725    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wbhkh\" (UniqueName: \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.646362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.646321    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.646637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.646579    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hwq4\" (UniqueName: \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.646795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.646776    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7spv\" (UniqueName: \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.646910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.646893    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d47th\" (UniqueName: \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.647031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.647018    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.647187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.647173    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.648081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.648057    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.648925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.648903    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.650445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.650408    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.712605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.711828    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hwq4\" (UniqueName: \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:00.731608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.715740    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d47th\" (UniqueName: \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:00.731608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.716413    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7spv\" (UniqueName: \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\") pod \"maas-api-key-cleanup-29889960-lvrng\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.731608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.725431    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:00.775657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.770872    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:00.800221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:00.799973    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:01.020609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.011946    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:01.278069 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:01.278035    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827cf109_6c12_4811_8b20_1f82ec834f60.slice/crio-11bb508ec88b45139e1e8f8567acf4ea8bd833db2c4363e072537f35063079e2 WatchSource:0}: Error finding container 11bb508ec88b45139e1e8f8567acf4ea8bd833db2c4363e072537f35063079e2: Status 404 returned error can't find the container with id 11bb508ec88b45139e1e8f8567acf4ea8bd833db2c4363e072537f35063079e2
Oct 30 22:00:01.283191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.283170    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:01.368735 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:01.368702    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29656b9_eea4_436e_8e39_51d04d7e81cd.slice/crio-8a357d5a4f6e7f20543f45c74565260a26c98362ae0045b699181731bc105b8d WatchSource:0}: Error finding container 8a357d5a4f6e7f20543f45c74565260a26c98362ae0045b699181731bc105b8d: Status 404 returned error can't find the container with id 8a357d5a4f6e7f20543f45c74565260a26c98362ae0045b699181731bc105b8d
Oct 30 22:00:01.373226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.373202    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:01.380629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.380579    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerStarted","Data":"11bb508ec88b45139e1e8f8567acf4ea8bd833db2c4363e072537f35063079e2"}
Oct 30 22:00:01.381942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.381915    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerStarted","Data":"8a357d5a4f6e7f20543f45c74565260a26c98362ae0045b699181731bc105b8d"}
Oct 30 22:00:01.401342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.400408    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:01.435389 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:01.435350    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7629372_1938_4983_8ade_171e14941ad3.slice/crio-8d09e2389845d7153003ca7015a7bba3625ea701bde322cae4f72b046331ca1d WatchSource:0}: Error finding container 8d09e2389845d7153003ca7015a7bba3625ea701bde322cae4f72b046331ca1d: Status 404 returned error can't find the container with id 8d09e2389845d7153003ca7015a7bba3625ea701bde322cae4f72b046331ca1d
Oct 30 22:00:01.440271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:01.438896    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:02.395561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.395521    4882 generic.go:356] "Generic (PLEG): container finished" podID="b7629372-1938-4983-8ade-171e14941ad3" containerID="5055e3e9d154c0ec9b9eb60142f71191aad5a5295e2b4038d0b3739650a91cab" exitCode=60
Oct 30 22:00:02.395827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.395627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerDied","Data":"5055e3e9d154c0ec9b9eb60142f71191aad5a5295e2b4038d0b3739650a91cab"}
Oct 30 22:00:02.395827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.395721    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerStarted","Data":"8d09e2389845d7153003ca7015a7bba3625ea701bde322cae4f72b046331ca1d"}
Oct 30 22:00:02.396070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.396053    4882 scope.go:122] "RemoveContainer" containerID="5055e3e9d154c0ec9b9eb60142f71191aad5a5295e2b4038d0b3739650a91cab"
Oct 30 22:00:02.397903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.397878    4882 generic.go:356] "Generic (PLEG): container finished" podID="827cf109-6c12-4811-8b20-1f82ec834f60" containerID="6053be9759b1d554a9832dceff6a4687b8fc8476d1107aed6c90fa9d9c512d8c" exitCode=60
Oct 30 22:00:02.397988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.397963    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerDied","Data":"6053be9759b1d554a9832dceff6a4687b8fc8476d1107aed6c90fa9d9c512d8c"}
Oct 30 22:00:02.398284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.398262    4882 scope.go:122] "RemoveContainer" containerID="6053be9759b1d554a9832dceff6a4687b8fc8476d1107aed6c90fa9d9c512d8c"
Oct 30 22:00:02.400365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.399718    4882 generic.go:356] "Generic (PLEG): container finished" podID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerID="59bf78936f98c376f51aad44f90c0c14142cc7612dd0365ac2c83bf23fe6dc28" exitCode=60
Oct 30 22:00:02.400365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.399802    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerDied","Data":"59bf78936f98c376f51aad44f90c0c14142cc7612dd0365ac2c83bf23fe6dc28"}
Oct 30 22:00:02.400365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.400033    4882 scope.go:122] "RemoveContainer" containerID="59bf78936f98c376f51aad44f90c0c14142cc7612dd0365ac2c83bf23fe6dc28"
Oct 30 22:00:02.401770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.401749    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerStarted","Data":"34912f66c95eb3d731c2b301e71be9193955139100454e70ee1146ebf48bd6ca"}
Oct 30 22:00:02.402080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.401798    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerStarted","Data":"ade5d4321c4c05363f211f9437ac7614f0b740a94771f08e96cff4ed205d30bc"}
Oct 30 22:00:02.760763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:02.760700    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" podStartSLOduration=2.760680301 podStartE2EDuration="2.760680301s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:02.755026459 +0000 UTC m=+1325.928350887" watchObservedRunningTime="2026-10-30 22:00:02.760680301 +0000 UTC m=+1325.934004728"
Oct 30 22:00:03.413553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.413511    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerStarted","Data":"bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"}
Oct 30 22:00:03.415647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.415620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerStarted","Data":"74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"}
Oct 30 22:00:03.417744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.417717    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerStarted","Data":"527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"}
Oct 30 22:00:03.434389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.433379    4882 generic.go:356] "Generic (PLEG): container finished" podID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerID="34912f66c95eb3d731c2b301e71be9193955139100454e70ee1146ebf48bd6ca" exitCode=60
Oct 30 22:00:03.434389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.433454    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerDied","Data":"34912f66c95eb3d731c2b301e71be9193955139100454e70ee1146ebf48bd6ca"}
Oct 30 22:00:03.434389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.433931    4882 scope.go:122] "RemoveContainer" containerID="34912f66c95eb3d731c2b301e71be9193955139100454e70ee1146ebf48bd6ca"
Oct 30 22:00:03.538293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.538244    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" podStartSLOduration=3.538231381 podStartE2EDuration="3.538231381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.536563302 +0000 UTC m=+1326.709887730" watchObservedRunningTime="2026-10-30 22:00:03.538231381 +0000 UTC m=+1326.711555813"
Oct 30 22:00:03.820690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.820488    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" podStartSLOduration=3.8204659100000002 podStartE2EDuration="3.82046591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.817469347 +0000 UTC m=+1326.990793787" watchObservedRunningTime="2026-10-30 22:00:03.82046591 +0000 UTC m=+1326.993790342"
Oct 30 22:00:03.823623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:03.823531    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" podStartSLOduration=3.8235112190000002 podStartE2EDuration="3.823511219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:03.694922465 +0000 UTC m=+1326.868246895" watchObservedRunningTime="2026-10-30 22:00:03.823511219 +0000 UTC m=+1326.996835648"
Oct 30 22:00:04.388983 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:04.388944    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37a55195_ad11_45cd_92e7_dcc6faf0d556.slice/crio-7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:00:04.444906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.444866    4882 generic.go:356] "Generic (PLEG): container finished" podID="b7629372-1938-4983-8ade-171e14941ad3" containerID="bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48" exitCode=60
Oct 30 22:00:04.445443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.444964    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerDied","Data":"bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"}
Oct 30 22:00:04.445443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.445039    4882 scope.go:122] "RemoveContainer" containerID="5055e3e9d154c0ec9b9eb60142f71191aad5a5295e2b4038d0b3739650a91cab"
Oct 30 22:00:04.445954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.445930    4882 scope.go:122] "RemoveContainer" containerID="bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"
Oct 30 22:00:04.446324 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:04.446296    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h_odh-ai-gateway-infra(b7629372-1938-4983-8ade-171e14941ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" podUID="b7629372-1938-4983-8ade-171e14941ad3"
Oct 30 22:00:04.448522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.448495    4882 generic.go:356] "Generic (PLEG): container finished" podID="827cf109-6c12-4811-8b20-1f82ec834f60" containerID="74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c" exitCode=60
Oct 30 22:00:04.448693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.448663    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerDied","Data":"74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"}
Oct 30 22:00:04.449012 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.448951    4882 scope.go:122] "RemoveContainer" containerID="74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"
Oct 30 22:00:04.449289 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:04.449263    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl_odh-ai-gateway-infra(827cf109-6c12-4811-8b20-1f82ec834f60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" podUID="827cf109-6c12-4811-8b20-1f82ec834f60"
Oct 30 22:00:04.450956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.450930    4882 generic.go:356] "Generic (PLEG): container finished" podID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerID="527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea" exitCode=60
Oct 30 22:00:04.451051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.451008    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerDied","Data":"527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"}
Oct 30 22:00:04.451341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.451322    4882 scope.go:122] "RemoveContainer" containerID="527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"
Oct 30 22:00:04.451675 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:04.451652    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-lvrng_odh-ai-gateway-infra(a29656b9-eea4-436e-8e39-51d04d7e81cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd"
Oct 30 22:00:04.453254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.453231    4882 generic.go:356] "Generic (PLEG): container finished" podID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerID="7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486" exitCode=60
Oct 30 22:00:04.453363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.453310    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerDied","Data":"7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486"}
Oct 30 22:00:04.453578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.453561    4882 scope.go:122] "RemoveContainer" containerID="7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486"
Oct 30 22:00:04.453926 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:04.453904    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj_odh-ai-gateway-infra(37a55195-ad11-45cd-92e7-dcc6faf0d556)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556"
Oct 30 22:00:04.471664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.471634    4882 scope.go:122] "RemoveContainer" containerID="6053be9759b1d554a9832dceff6a4687b8fc8476d1107aed6c90fa9d9c512d8c"
Oct 30 22:00:04.487175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.487146    4882 scope.go:122] "RemoveContainer" containerID="59bf78936f98c376f51aad44f90c0c14142cc7612dd0365ac2c83bf23fe6dc28"
Oct 30 22:00:04.503447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:04.503424    4882 scope.go:122] "RemoveContainer" containerID="34912f66c95eb3d731c2b301e71be9193955139100454e70ee1146ebf48bd6ca"
Oct 30 22:00:05.471231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:05.471191    4882 scope.go:122] "RemoveContainer" containerID="7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486"
Oct 30 22:00:05.471715 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:05.471565    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj_odh-ai-gateway-infra(37a55195-ad11-45cd-92e7-dcc6faf0d556)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556"
Oct 30 22:00:05.473414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:05.473373    4882 scope.go:122] "RemoveContainer" containerID="bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"
Oct 30 22:00:05.473726 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:05.473701    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h_odh-ai-gateway-infra(b7629372-1938-4983-8ade-171e14941ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" podUID="b7629372-1938-4983-8ade-171e14941ad3"
Oct 30 22:00:05.474961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:05.474941    4882 scope.go:122] "RemoveContainer" containerID="74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"
Oct 30 22:00:05.475198 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:05.475169    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl_odh-ai-gateway-infra(827cf109-6c12-4811-8b20-1f82ec834f60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" podUID="827cf109-6c12-4811-8b20-1f82ec834f60"
Oct 30 22:00:05.476544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:05.476528    4882 scope.go:122] "RemoveContainer" containerID="527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"
Oct 30 22:00:05.476778 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:05.476761    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-lvrng_odh-ai-gateway-infra(a29656b9-eea4-436e-8e39-51d04d7e81cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd"
Oct 30 22:00:13.936422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:13.936392    4882 scope.go:122] "RemoveContainer" containerID="bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"
Oct 30 22:00:14.543715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:14.543620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerStarted","Data":"1358a7f8a9aa55b42351c1b4169d6e8cc115cba2264ad89ad20af791abc5cc5d"}
Oct 30 22:00:14.936939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:14.936908    4882 scope.go:122] "RemoveContainer" containerID="7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486"
Oct 30 22:00:15.552985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.552899    4882 generic.go:356] "Generic (PLEG): container finished" podID="b7629372-1938-4983-8ade-171e14941ad3" containerID="1358a7f8a9aa55b42351c1b4169d6e8cc115cba2264ad89ad20af791abc5cc5d" exitCode=60
Oct 30 22:00:15.552985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.552961    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerDied","Data":"1358a7f8a9aa55b42351c1b4169d6e8cc115cba2264ad89ad20af791abc5cc5d"}
Oct 30 22:00:15.553198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.553021    4882 scope.go:122] "RemoveContainer" containerID="bc5623473869feda04dd1977f2314bdc386f2c7be1014b0d4c90c88296379d48"
Oct 30 22:00:15.553487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.553467    4882 scope.go:122] "RemoveContainer" containerID="1358a7f8a9aa55b42351c1b4169d6e8cc115cba2264ad89ad20af791abc5cc5d"
Oct 30 22:00:15.553816 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:15.553794    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h_odh-ai-gateway-infra(b7629372-1938-4983-8ade-171e14941ad3)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" podUID="b7629372-1938-4983-8ade-171e14941ad3"
Oct 30 22:00:15.555171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.555146    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerStarted","Data":"53212e3452da479d7e09232db3e63dbbd95e38f21f7c47b28d3224189ff9ffb0"}
Oct 30 22:00:15.682949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:15.682917    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:16.567002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.566968    4882 generic.go:356] "Generic (PLEG): container finished" podID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerID="53212e3452da479d7e09232db3e63dbbd95e38f21f7c47b28d3224189ff9ffb0" exitCode=60
Oct 30 22:00:16.567477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.567026    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerDied","Data":"53212e3452da479d7e09232db3e63dbbd95e38f21f7c47b28d3224189ff9ffb0"}
Oct 30 22:00:16.567597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.567528    4882 scope.go:122] "RemoveContainer" containerID="7178ef2450e9be2bfb6dde96f9d93ea14a2523df6821b8cced7d68baf8faf486"
Oct 30 22:00:16.567972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.567953    4882 scope.go:122] "RemoveContainer" containerID="53212e3452da479d7e09232db3e63dbbd95e38f21f7c47b28d3224189ff9ffb0"
Oct 30 22:00:16.568275 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:16.568254    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj_odh-ai-gateway-infra(37a55195-ad11-45cd-92e7-dcc6faf0d556)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556"
Oct 30 22:00:16.601995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.601959    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"]
Oct 30 22:00:16.607452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.607428    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.660506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.660469    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:00:16.669302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.669263    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.669467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.669397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppz2g\" (UniqueName: \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.669467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.669451    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.689381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.689317    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"]
Oct 30 22:00:16.740046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.740017    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:16.770567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770538    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\" (UniqueName: \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\") pod \"b7629372-1938-4983-8ade-171e14941ad3\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") "
Oct 30 22:00:16.770752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770621    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\") pod \"b7629372-1938-4983-8ade-171e14941ad3\" (UID: \"b7629372-1938-4983-8ade-171e14941ad3\") "
Oct 30 22:00:16.770838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770823    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.770921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770905    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppz2g\" (UniqueName: \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.770982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770933    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca" pod "b7629372-1938-4983-8ade-171e14941ad3" (UID: "b7629372-1938-4983-8ade-171e14941ad3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:16.770982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.770946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.771200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.771183    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b7629372-1938-4983-8ade-171e14941ad3-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:16.771630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.771613    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.772976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.772936    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th" pod "b7629372-1938-4983-8ade-171e14941ad3" (UID: "b7629372-1938-4983-8ade-171e14941ad3"). InnerVolumeSpecName "kube-api-access-d47th". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:16.773169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.773149    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.800738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.800708    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:16.836700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.836663    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppz2g\" (UniqueName: \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\") pod \"maas-api-revoke-sub-01ecd803-5x49c\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.872754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.872715    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d47th\" (UniqueName: \"kubernetes.io/projected/b7629372-1938-4983-8ade-171e14941ad3-kube-api-access-d47th\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:16.923116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.923080    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:16.936223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.936204    4882 scope.go:122] "RemoveContainer" containerID="74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"
Oct 30 22:00:16.936377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:16.936356    4882 scope.go:122] "RemoveContainer" containerID="527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"
Oct 30 22:00:17.307843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.307809    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"]
Oct 30 22:00:17.327113 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:17.327082    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8213404_5b59_4fb2_b200_274500dd4dc0.slice/crio-4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e WatchSource:0}: Error finding container 4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e: Status 404 returned error can't find the container with id 4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e
Oct 30 22:00:17.577941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.577915    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"
Oct 30 22:00:17.578337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.577912    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h" event={"ID":"b7629372-1938-4983-8ade-171e14941ad3","Type":"ContainerDied","Data":"8d09e2389845d7153003ca7015a7bba3625ea701bde322cae4f72b046331ca1d"}
Oct 30 22:00:17.578337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.578063    4882 scope.go:122] "RemoveContainer" containerID="1358a7f8a9aa55b42351c1b4169d6e8cc115cba2264ad89ad20af791abc5cc5d"
Oct 30 22:00:17.580491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.580461    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerStarted","Data":"6ae07b5c9dc14891e44eb345ac8e25a594c36f860d99f4985cb53e79090c7991"}
Oct 30 22:00:17.583242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.583215    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerStarted","Data":"f55be8e2cf4098de51685c7b745b8a38d1084014bb1b73edb45f0ab832ed7557"}
Oct 30 22:00:17.584891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.584845    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c" event={"ID":"c8213404-5b59-4fb2-b200-274500dd4dc0","Type":"ContainerStarted","Data":"4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e"}
Oct 30 22:00:17.748195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.748160    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:17.782886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.782857    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\") pod \"37a55195-ad11-45cd-92e7-dcc6faf0d556\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") "
Oct 30 22:00:17.783064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.783049    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\" (UniqueName: \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\") pod \"37a55195-ad11-45cd-92e7-dcc6faf0d556\" (UID: \"37a55195-ad11-45cd-92e7-dcc6faf0d556\") "
Oct 30 22:00:17.783273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.783245    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca" pod "37a55195-ad11-45cd-92e7-dcc6faf0d556" (UID: "37a55195-ad11-45cd-92e7-dcc6faf0d556"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:17.783465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.783450    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/37a55195-ad11-45cd-92e7-dcc6faf0d556-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:17.785202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.785140    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4" pod "37a55195-ad11-45cd-92e7-dcc6faf0d556" (UID: "37a55195-ad11-45cd-92e7-dcc6faf0d556"). InnerVolumeSpecName "kube-api-access-9hwq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:17.885010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:17.884977    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hwq4\" (UniqueName: \"kubernetes.io/projected/37a55195-ad11-45cd-92e7-dcc6faf0d556-kube-api-access-9hwq4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:18.042911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.042829    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:18.050066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.050034    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889960-gh49h"]
Oct 30 22:00:18.596090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.596058    4882 generic.go:356] "Generic (PLEG): container finished" podID="827cf109-6c12-4811-8b20-1f82ec834f60" containerID="6ae07b5c9dc14891e44eb345ac8e25a594c36f860d99f4985cb53e79090c7991" exitCode=60
Oct 30 22:00:18.596575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.596129    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerDied","Data":"6ae07b5c9dc14891e44eb345ac8e25a594c36f860d99f4985cb53e79090c7991"}
Oct 30 22:00:18.596575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.596184    4882 scope.go:122] "RemoveContainer" containerID="74da289eba1285d0f45cc7b5e46b2098b1abfb269765630234839d946956439c"
Oct 30 22:00:18.596575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.596551    4882 scope.go:122] "RemoveContainer" containerID="6ae07b5c9dc14891e44eb345ac8e25a594c36f860d99f4985cb53e79090c7991"
Oct 30 22:00:18.596914 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:18.596890    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl_odh-ai-gateway-infra(827cf109-6c12-4811-8b20-1f82ec834f60)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" podUID="827cf109-6c12-4811-8b20-1f82ec834f60"
Oct 30 22:00:18.598399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.598378    4882 generic.go:356] "Generic (PLEG): container finished" podID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerID="f55be8e2cf4098de51685c7b745b8a38d1084014bb1b73edb45f0ab832ed7557" exitCode=60
Oct 30 22:00:18.598502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.598446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerDied","Data":"f55be8e2cf4098de51685c7b745b8a38d1084014bb1b73edb45f0ab832ed7557"}
Oct 30 22:00:18.598830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.598811    4882 scope.go:122] "RemoveContainer" containerID="f55be8e2cf4098de51685c7b745b8a38d1084014bb1b73edb45f0ab832ed7557"
Oct 30 22:00:18.599111 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:18.599086    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889960-lvrng_odh-ai-gateway-infra(a29656b9-eea4-436e-8e39-51d04d7e81cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd"
Oct 30 22:00:18.600215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.600197    4882 generic.go:356] "Generic (PLEG): container finished" podID="c8213404-5b59-4fb2-b200-274500dd4dc0" containerID="5b0c4d2b7ae2c36370c5c231d0f51ae2ea7821174ce75e4c929ac2aa66f7a7f6" exitCode=0
Oct 30 22:00:18.600317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.600279    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c" event={"ID":"c8213404-5b59-4fb2-b200-274500dd4dc0","Type":"ContainerDied","Data":"5b0c4d2b7ae2c36370c5c231d0f51ae2ea7821174ce75e4c929ac2aa66f7a7f6"}
Oct 30 22:00:18.601815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.601796    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"
Oct 30 22:00:18.601815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.601800    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj" event={"ID":"37a55195-ad11-45cd-92e7-dcc6faf0d556","Type":"ContainerDied","Data":"ade5d4321c4c05363f211f9437ac7614f0b740a94771f08e96cff4ed205d30bc"}
Oct 30 22:00:18.611623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.611574    4882 scope.go:122] "RemoveContainer" containerID="527d44adbc52a2588cbe9b4cb9d4b3100ea4eba31256cc4535b3787448fbb3ea"
Oct 30 22:00:18.626051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.626029    4882 scope.go:122] "RemoveContainer" containerID="53212e3452da479d7e09232db3e63dbbd95e38f21f7c47b28d3224189ff9ffb0"
Oct 30 22:00:18.831094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.831058    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:18.954832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.954751    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7629372-1938-4983-8ade-171e14941ad3" path="/var/lib/kubelet/pods/b7629372-1938-4983-8ade-171e14941ad3/volumes"
Oct 30 22:00:18.955099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:18.955084    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:19.007910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.007878    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:19.024808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.024781    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889960-8s6nj"]
Oct 30 22:00:19.855694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.855663    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:19.859259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.859235    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:19.864063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.864040    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:19.909014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.908977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\") pod \"a29656b9-eea4-436e-8e39-51d04d7e81cd\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") "
Oct 30 22:00:19.909228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909037    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\") pod \"c8213404-5b59-4fb2-b200-274500dd4dc0\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") "
Oct 30 22:00:19.909228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909096    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\") pod \"827cf109-6c12-4811-8b20-1f82ec834f60\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") "
Oct 30 22:00:19.909228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909152    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\" (UniqueName: \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\") pod \"827cf109-6c12-4811-8b20-1f82ec834f60\" (UID: \"827cf109-6c12-4811-8b20-1f82ec834f60\") "
Oct 30 22:00:19.909228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909200    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\" (UniqueName: \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\") pod \"a29656b9-eea4-436e-8e39-51d04d7e81cd\" (UID: \"a29656b9-eea4-436e-8e39-51d04d7e81cd\") "
Oct 30 22:00:19.909468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909269    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\") pod \"c8213404-5b59-4fb2-b200-274500dd4dc0\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") "
Oct 30 22:00:19.909468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909371    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\" (UniqueName: \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\") pod \"c8213404-5b59-4fb2-b200-274500dd4dc0\" (UID: \"c8213404-5b59-4fb2-b200-274500dd4dc0\") "
Oct 30 22:00:19.909566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909470    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca" pod "a29656b9-eea4-436e-8e39-51d04d7e81cd" (UID: "a29656b9-eea4-436e-8e39-51d04d7e81cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.909566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909492    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca" pod "c8213404-5b59-4fb2-b200-274500dd4dc0" (UID: "c8213404-5b59-4fb2-b200-274500dd4dc0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.909566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.909526    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca" pod "827cf109-6c12-4811-8b20-1f82ec834f60" (UID: "827cf109-6c12-4811-8b20-1f82ec834f60"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:00:19.910026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.910001    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a29656b9-eea4-436e-8e39-51d04d7e81cd-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.910187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.910031    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.910187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.910046    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/827cf109-6c12-4811-8b20-1f82ec834f60-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:19.912058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.912033    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca" pod "c8213404-5b59-4fb2-b200-274500dd4dc0" (UID: "c8213404-5b59-4fb2-b200-274500dd4dc0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:00:19.912158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.912127    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv" pod "a29656b9-eea4-436e-8e39-51d04d7e81cd" (UID: "a29656b9-eea4-436e-8e39-51d04d7e81cd"). InnerVolumeSpecName "kube-api-access-c7spv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.912659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.912638    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g" pod "c8213404-5b59-4fb2-b200-274500dd4dc0" (UID: "c8213404-5b59-4fb2-b200-274500dd4dc0"). InnerVolumeSpecName "kube-api-access-ppz2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:19.912778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:19.912670    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh" pod "827cf109-6c12-4811-8b20-1f82ec834f60" (UID: "827cf109-6c12-4811-8b20-1f82ec834f60"). InnerVolumeSpecName "kube-api-access-wbhkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:00:20.011262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.011226    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wbhkh\" (UniqueName: \"kubernetes.io/projected/827cf109-6c12-4811-8b20-1f82ec834f60-kube-api-access-wbhkh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.011262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.011260    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7spv\" (UniqueName: \"kubernetes.io/projected/a29656b9-eea4-436e-8e39-51d04d7e81cd-kube-api-access-c7spv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.011468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.011273    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c8213404-5b59-4fb2-b200-274500dd4dc0-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.011468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.011287    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppz2g\" (UniqueName: \"kubernetes.io/projected/c8213404-5b59-4fb2-b200-274500dd4dc0-kube-api-access-ppz2g\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:00:20.625904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.625869    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"
Oct 30 22:00:20.625904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.625890    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c" event={"ID":"c8213404-5b59-4fb2-b200-274500dd4dc0","Type":"ContainerDied","Data":"4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e"}
Oct 30 22:00:20.626107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.625937    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4205707dd8694b8c86878266396879b4ad1eb89f6e1e668618d6e538c9ea077e"
Oct 30 22:00:20.627441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.627422    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"
Oct 30 22:00:20.627441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.627424    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl" event={"ID":"827cf109-6c12-4811-8b20-1f82ec834f60","Type":"ContainerDied","Data":"11bb508ec88b45139e1e8f8567acf4ea8bd833db2c4363e072537f35063079e2"}
Oct 30 22:00:20.627668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.627481    4882 scope.go:122] "RemoveContainer" containerID="6ae07b5c9dc14891e44eb345ac8e25a594c36f860d99f4985cb53e79090c7991"
Oct 30 22:00:20.629316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.629292    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng" event={"ID":"a29656b9-eea4-436e-8e39-51d04d7e81cd","Type":"ContainerDied","Data":"8a357d5a4f6e7f20543f45c74565260a26c98362ae0045b699181731bc105b8d"}
Oct 30 22:00:20.629399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.629318    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"
Oct 30 22:00:20.638836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.638810    4882 scope.go:122] "RemoveContainer" containerID="f55be8e2cf4098de51685c7b745b8a38d1084014bb1b73edb45f0ab832ed7557"
Oct 30 22:00:20.811095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.809753    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:20.811095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.809808    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889960-lvrng"]
Oct 30 22:00:20.816679 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:20.816648    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:00:20.943620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.943530    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" path="/var/lib/kubelet/pods/37a55195-ad11-45cd-92e7-dcc6faf0d556/volumes"
Oct 30 22:00:20.944055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.944039    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" path="/var/lib/kubelet/pods/a29656b9-eea4-436e-8e39-51d04d7e81cd/volumes"
Oct 30 22:00:20.993228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.993182    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:20.997407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:20.997369    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889960-bjkgl"]
Oct 30 22:00:22.943070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:22.943037    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" path="/var/lib/kubelet/pods/827cf109-6c12-4811-8b20-1f82ec834f60/volumes"
Oct 30 22:00:24.797631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.797580    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:00:24.798221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798205    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798222    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798231    4882 cpu_manager.go:414] "containerMap: removing container" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798251    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798264    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798270    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798277    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798282    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798293    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c8213404-5b59-4fb2-b200-274500dd4dc0" containerName="revoke-keys"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798298    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8213404-5b59-4fb2-b200-274500dd4dc0" containerName="revoke-keys"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798306    4882 cpu_manager.go:414] "containerMap: removing container" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798310    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798327    4882 cpu_manager.go:414] "containerMap: removing container" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798331    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798341    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798346    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798355    4882 cpu_manager.go:414] "containerMap: removing container" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798359    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798368    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798373    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798462    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798471    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798478    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798485    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798492    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798501    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798508    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798515    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798521    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29656b9-eea4-436e-8e39-51d04d7e81cd" containerName="cleanup"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798526    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8213404-5b59-4fb2-b200-274500dd4dc0" containerName="revoke-keys"
Oct 30 22:00:24.798937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.798534    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:24.803690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.803665    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876078    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876156    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876257    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876405    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876458    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876511    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vmwp\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876682    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876715    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.876754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.876741    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.957211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.957172    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:00:24.978595 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978548    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978602    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978624    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978682    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978728    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978758    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978862    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978903    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.978975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.978936    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vmwp\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.979230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.979201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.979230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.979228    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.979536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.979508    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.979670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.979538    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.979670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.979650    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.981974 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.981955    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:24.982296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:24.982275    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:25.041760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.041729    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:25.041904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.041771    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vmwp\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:25.118540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.118490    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:25.546343 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:25.546292    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f7a95fd_d284_4220_a0c1_80cff2c18524.slice/crio-367f32393963ea17df14ef5be2f975d895b34a66936bc3344ba73e60bf1b5784 WatchSource:0}: Error finding container 367f32393963ea17df14ef5be2f975d895b34a66936bc3344ba73e60bf1b5784: Status 404 returned error can't find the container with id 367f32393963ea17df14ef5be2f975d895b34a66936bc3344ba73e60bf1b5784
Oct 30 22:00:25.549213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.549174    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:00:25.549305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.549259    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:00:25.549342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.549312    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:00:25.550120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.550098    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:00:25.671072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.671022    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" event={"ID":"8f7a95fd-d284-4220-a0c1-80cff2c18524","Type":"ContainerStarted","Data":"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"}
Oct 30 22:00:25.671263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.671090    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" event={"ID":"8f7a95fd-d284-4220-a0c1-80cff2c18524","Type":"ContainerStarted","Data":"367f32393963ea17df14ef5be2f975d895b34a66936bc3344ba73e60bf1b5784"}
Oct 30 22:00:25.820575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:25.820278    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podStartSLOduration=1.820255822 podStartE2EDuration="1.820255822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:25.794808474 +0000 UTC m=+1348.968132904" watchObservedRunningTime="2026-10-30 22:00:25.820255822 +0000 UTC m=+1348.993580253"
Oct 30 22:00:26.119656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:26.119619    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:26.121445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:26.121414    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:26.121626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:26.121481    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:27.119650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:27.119579    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:27.120150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:27.119686    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:27.564308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:27.564222    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:00:28.119254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:28.119223    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:28.119441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:28.119283    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:29.119580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:29.119539    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:29.119997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:29.119653    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:30.119278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:30.119238    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:30.119467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:30.119311    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:31.119527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:31.119491    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:31.120183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:31.120132    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:32.125610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:32.122106    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:32.125610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:32.122189    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:33.118955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:33.118918    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused" start-of-body=
Oct 30 22:00:33.119223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:33.118989    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.98:15021/healthz/ready\": dial tcp 10.132.0.98:15021: connect: connection refused"
Oct 30 22:00:34.883796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:34.883761    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:34.884271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:34.884100    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:34.885195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:34.885172    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:00:38.110715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.110623    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:00:38.111461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111438    4882 cpu_manager.go:414] "containerMap: removing container" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:38.111565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111462    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:38.111565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111481    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:38.111565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111489    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7629372-1938-4983-8ade-171e14941ad3" containerName="cleanup"
Oct 30 22:00:38.111565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111537    4882 cpu_manager.go:414] "containerMap: removing container" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:38.111565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111546    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:38.111865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111739    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="827cf109-6c12-4811-8b20-1f82ec834f60" containerName="cleanup"
Oct 30 22:00:38.111865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.111755    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a55195-ad11-45cd-92e7-dcc6faf0d556" containerName="cleanup"
Oct 30 22:00:38.116622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.116572    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.160478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.160674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160631    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.160730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160701    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.160793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160764    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blkr5\" (UniqueName: \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.160865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160849    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.161005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.160990    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.170347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.170316    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:00:38.216623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.216577    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-3c4802\"/\"distinct-2-w1-3c4802-kserve-self-signed-certs\""
Oct 30 22:00:38.262810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.262767    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.263080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.263007    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.263442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.263216    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.263442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.263332    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.263879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.263847    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blkr5\" (UniqueName: \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.264049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.263898    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.265882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.265821    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.268210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.267009    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.268210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.267141    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.268723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.268691    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.271451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.271424    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.341511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.341482    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-blkr5\" (UniqueName: \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\") pod \"distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") " pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.430275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.430169    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:38.810882 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:00:38.810850    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod59948204_a285_4476_8079_17089fdddef9.slice/crio-613e046b3c5ffe506771dc3b0ffdbc6978249b4a404f0e3d88bd5f81b9e6b4eb WatchSource:0}: Error finding container 613e046b3c5ffe506771dc3b0ffdbc6978249b4a404f0e3d88bd5f81b9e6b4eb: Status 404 returned error can't find the container with id 613e046b3c5ffe506771dc3b0ffdbc6978249b4a404f0e3d88bd5f81b9e6b4eb
Oct 30 22:00:38.814364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.814337    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:00:38.837016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:38.836988    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" event={"ID":"59948204-a285-4476-8079-17089fdddef9","Type":"ContainerStarted","Data":"613e046b3c5ffe506771dc3b0ffdbc6978249b4a404f0e3d88bd5f81b9e6b4eb"}
Oct 30 22:00:39.846134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:39.846094    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" event={"ID":"59948204-a285-4476-8079-17089fdddef9","Type":"ContainerStarted","Data":"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"}
Oct 30 22:00:40.006172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:40.006116    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" podStartSLOduration=2.006102999 podStartE2EDuration="2.006102999s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:00:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:00:39.966554422 +0000 UTC m=+1363.139878848" watchObservedRunningTime="2026-10-30 22:00:40.006102999 +0000 UTC m=+1363.179427425"
Oct 30 22:00:40.430948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:40.430913    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:48.445610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:48.440546    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:48.515395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:48.515328    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:49.104027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:49.103987    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:00:50.809520 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:00:50.809478    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:00:57.389333 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.389298    4882 scope.go:122] "RemoveContainer" containerID="46df8b94db7946ca905c0cb70967393e57351f59a029cd1ee7506bac0c592ee4"
Oct 30 22:00:57.399118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.399094    4882 scope.go:122] "RemoveContainer" containerID="5f0c2a6c114d5d4efdd68d01044cdce213d0a637a02425fbacb930816eb2c878"
Oct 30 22:00:57.408664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.408638    4882 scope.go:122] "RemoveContainer" containerID="9bb6897164fe8f9c259729c34bae9e409704bd448caee3723a24dad79da52159"
Oct 30 22:00:57.420130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.420106    4882 scope.go:122] "RemoveContainer" containerID="a990a0f94ab955e99a93951c432798264297e37890bf72e979c903ed9b7c6d00"
Oct 30 22:00:57.429414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.429396    4882 scope.go:122] "RemoveContainer" containerID="be341226a7052dd07cc251819f062fa1fff35142736d9b77d436c4a2c929df0d"
Oct 30 22:00:57.438501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.438478    4882 scope.go:122] "RemoveContainer" containerID="3ca38c406be1ffca38848142b5100ed4c86e7961de90cf7a5afa258049015ccc"
Oct 30 22:00:57.448090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:00:57.448067    4882 scope.go:122] "RemoveContainer" containerID="34c6fbc71f44d6a2892ea7771466202d2d646deb972592724c6d43ad0ccc9ec3"
Oct 30 22:01:09.025376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:09.025339    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:01:13.105625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.105575    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:01:13.110050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.110031    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.155622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.155582    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:01:13.159861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.159843    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.208743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.208707    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:01:13.214566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214534    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.214752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214632    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.214752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.214839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214800    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.214839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjp28\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.214923 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.214859    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.215078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.215037    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.215161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.215145    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.215214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.215185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.266461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.266425    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:01:13.316545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.316545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316549    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.316780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316674    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.316780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316707    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.316780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316739    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.316780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316760    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.316904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316884    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.316941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316922    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.316987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjp28\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.316987    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317043    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flm86\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.317134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317078    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.317134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317096    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317122    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317244    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317293    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.317353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317331    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317351    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317395    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.317456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317411    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.317557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317534    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.317612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.317563    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.318072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.318045    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.319151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.319132    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.319541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.319526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.381027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.380958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.381188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.381162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjp28\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.418708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418672    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418711    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418731    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418793    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flm86\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418814    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.418985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.418908    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419012    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419127    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419158    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419254    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419281    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.419663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.419644    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.421211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.421189    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.421407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.421389    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.424289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.424270    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:13.486548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.486506    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.486872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.486849    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flm86\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.774619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.774518    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:13.857808 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:01:13.857780    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7194050e_8c66_4e18_8d67_371a4622757d.slice/crio-e7529a9926e79b7e4cf953135c884219a739da906c2d253843f8cf0e9ea28655 WatchSource:0}: Error finding container e7529a9926e79b7e4cf953135c884219a739da906c2d253843f8cf0e9ea28655: Status 404 returned error can't find the container with id e7529a9926e79b7e4cf953135c884219a739da906c2d253843f8cf0e9ea28655
Oct 30 22:01:13.860788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.860744    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:13.860912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.860835    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:13.860912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.860878    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:13.862568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:13.862538    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:01:14.166883 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:01:14.166848    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae462eed_1dab_45f5_8af4_eeede962eea9.slice/crio-c3b2cfc32d1bc7adbcd56caf5aabb3a9c6b4a46e22d4ba3d9deedd90df92c678 WatchSource:0}: Error finding container c3b2cfc32d1bc7adbcd56caf5aabb3a9c6b4a46e22d4ba3d9deedd90df92c678: Status 404 returned error can't find the container with id c3b2cfc32d1bc7adbcd56caf5aabb3a9c6b4a46e22d4ba3d9deedd90df92c678
Oct 30 22:01:14.170154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.170116    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:14.170248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.170189    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:14.170248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.170217    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:14.174258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.174234    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:01:14.213620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.213566    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" event={"ID":"ae462eed-1dab-45f5-8af4-eeede962eea9","Type":"ContainerStarted","Data":"c3b2cfc32d1bc7adbcd56caf5aabb3a9c6b4a46e22d4ba3d9deedd90df92c678"}
Oct 30 22:01:14.216127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.216095    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" event={"ID":"7194050e-8c66-4e18-8d67-371a4622757d","Type":"ContainerStarted","Data":"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"}
Oct 30 22:01:14.216242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.216158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" event={"ID":"7194050e-8c66-4e18-8d67-371a4622757d","Type":"ContainerStarted","Data":"e7529a9926e79b7e4cf953135c884219a739da906c2d253843f8cf0e9ea28655"}
Oct 30 22:01:14.341875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.341799    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podStartSLOduration=1.341780037 podStartE2EDuration="1.341780037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:14.338154132 +0000 UTC m=+1397.511478558" watchObservedRunningTime="2026-10-30 22:01:14.341780037 +0000 UTC m=+1397.515104467"
Oct 30 22:01:14.428670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.424741    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:14.439615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.435731    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:14.439615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:14.435797    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:15.224417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.224382    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" event={"ID":"ae462eed-1dab-45f5-8af4-eeede962eea9","Type":"ContainerStarted","Data":"149f6d50b4a093a84de57301dc7e0019d509ed4530b051d70891b7a11647e1b5"}
Oct 30 22:01:15.436611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.429054    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:15.436611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.429154    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:15.774820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.774770    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:15.776937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.776902    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:15.777100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:15.776965    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:16.429614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:16.428973    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:16.429614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:16.429043    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:16.775337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:16.775233    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:16.775337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:16.775297    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:17.425988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:17.425949    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:17.426314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:17.426275    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:17.776671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:17.776101    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:17.776671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:17.776179    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:18.442610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:18.442002    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:18.442610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:18.442068    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:18.776111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:18.776010    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:18.776429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:18.776398    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:19.429037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:19.428993    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:19.429651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:19.429614    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:19.778677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:19.775908    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:19.778677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:19.775970    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:20.425837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:20.425685    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused" start-of-body=
Oct 30 22:01:20.425837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:20.425754    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.100:15021/healthz/ready\": dial tcp 10.132.0.100:15021: connect: connection refused"
Oct 30 22:01:20.776092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:20.775997    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused" start-of-body=
Oct 30 22:01:20.776092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:20.776064    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.101:15021/healthz/ready\": dial tcp 10.132.0.101:15021: connect: connection refused"
Oct 30 22:01:20.812679 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:01:20.812638    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:01:21.429630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.429600    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:21.430200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.430162    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:21.432263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.432239    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:01:21.660975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.660917    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podStartSLOduration=8.660902562 podStartE2EDuration="8.660902562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:15.357161389 +0000 UTC m=+1398.530485817" watchObservedRunningTime="2026-10-30 22:01:21.660902562 +0000 UTC m=+1404.834226987"
Oct 30 22:01:21.779920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.779840    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:21.780362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.780220    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:21.781141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:21.781120    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:01:44.998029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:44.997993    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:01:45.003083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.003062    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.034782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.034745    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035038    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035083    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc4s7\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035115    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035189    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035311    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035437    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035462    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.035571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.035548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.056066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.056039    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-26743a-openshift-default-dockercfg-lddtx\""
Oct 30 22:01:45.112887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.112846    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:01:45.136538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.136505    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.136813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.136787    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc4s7\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.136921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.136908    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137058    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137162    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137279    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137365    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137495    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.137766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.137750    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.139189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.139158    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.139877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.139854    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.140248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.140228    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.157382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.153805    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.162920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.158260    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.163120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.162850    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.204651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.204615    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.207353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.207317    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.207554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.207526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc4s7\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.318314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.318213    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:45.744941 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:01:45.744904    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82fec621_76db_4ce6_b6cf_47522601622a.slice/crio-cc817a718ab1d021761baa171d4651bdd6077742c6718b47b443879297147568 WatchSource:0}: Error finding container cc817a718ab1d021761baa171d4651bdd6077742c6718b47b443879297147568: Status 404 returned error can't find the container with id cc817a718ab1d021761baa171d4651bdd6077742c6718b47b443879297147568
Oct 30 22:01:45.747723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.747680    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:45.747825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.747769    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:45.747825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.747814    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:01:45.749154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:45.749129    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:01:46.485281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:46.485247    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" event={"ID":"82fec621-76db-4ce6-b6cf-47522601622a","Type":"ContainerStarted","Data":"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"}
Oct 30 22:01:46.485715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:46.485300    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" event={"ID":"82fec621-76db-4ce6-b6cf-47522601622a","Type":"ContainerStarted","Data":"cc817a718ab1d021761baa171d4651bdd6077742c6718b47b443879297147568"}
Oct 30 22:01:46.713006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:46.712954    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podStartSLOduration=2.712938203 podStartE2EDuration="2.712938203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:46.660504704 +0000 UTC m=+1429.833829129" watchObservedRunningTime="2026-10-30 22:01:46.712938203 +0000 UTC m=+1429.886262628"
Oct 30 22:01:47.319134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:47.319084    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:47.320793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:47.320755    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:47.320922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:47.320823    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:48.319000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:48.318960    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:48.319423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:48.319017    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:49.082055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.082020    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"]
Oct 30 22:01:49.086513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.086495    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.155174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.148730    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:49.158881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.158847    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"]
Oct 30 22:01:49.190377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.190326    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcqp4\" (UniqueName: \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.190602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.190392    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.190695 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.190609    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.196496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.196467    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:01:49.291884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.291847    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.292054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.292021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcqp4\" (UniqueName: \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.292093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.292063    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.292543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.292519    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.294560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.294532    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.319187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.319152    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:49.319568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.319226    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:49.356247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.356166    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcqp4\" (UniqueName: \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\") pod \"maas-api-revoke-sub-829c419d-5bvs6\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.400829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.400795    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:49.503608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.503560    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:01:49.805072 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:01:49.805017    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a73c1eb_e967_4c64_8b8b_5530e7fb190a.slice/crio-ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5 WatchSource:0}: Error finding container ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5: Status 404 returned error can't find the container with id ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5
Oct 30 22:01:49.809916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:49.809869    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"]
Oct 30 22:01:50.319498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:50.319465    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:50.319976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:50.319516    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:50.524251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:50.524157    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6" event={"ID":"6a73c1eb-e967-4c64-8b8b-5530e7fb190a","Type":"ContainerStarted","Data":"492115f2a2ae8078f7165e3810e6ef3f295f0335362f497f394cd142fd985bba"}
Oct 30 22:01:50.524251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:50.524228    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6" event={"ID":"6a73c1eb-e967-4c64-8b8b-5530e7fb190a","Type":"ContainerStarted","Data":"ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5"}
Oct 30 22:01:50.823655 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:01:50.823484    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:01:51.327274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:51.327229    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:51.327925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:51.327886    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:51.575611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:51.567142    4882 generic.go:356] "Generic (PLEG): container finished" podID="6a73c1eb-e967-4c64-8b8b-5530e7fb190a" containerID="492115f2a2ae8078f7165e3810e6ef3f295f0335362f497f394cd142fd985bba" exitCode=0
Oct 30 22:01:51.575611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:51.567218    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6" event={"ID":"6a73c1eb-e967-4c64-8b8b-5530e7fb190a","Type":"ContainerDied","Data":"492115f2a2ae8078f7165e3810e6ef3f295f0335362f497f394cd142fd985bba"}
Oct 30 22:01:52.319431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.319387    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused" start-of-body=
Oct 30 22:01:52.319642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.319458    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.102:15021/healthz/ready\": dial tcp 10.132.0.102:15021: connect: connection refused"
Oct 30 22:01:52.750189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.750161    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:52.807951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.807916    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\") pod \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") "
Oct 30 22:01:52.808129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.808088    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\") pod \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") "
Oct 30 22:01:52.808201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.808186    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\" (UniqueName: \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\") pod \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\" (UID: \"6a73c1eb-e967-4c64-8b8b-5530e7fb190a\") "
Oct 30 22:01:52.808507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.808478    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca" pod "6a73c1eb-e967-4c64-8b8b-5530e7fb190a" (UID: "6a73c1eb-e967-4c64-8b8b-5530e7fb190a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:01:52.808725 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.808706    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.810138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.810107    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca" pod "6a73c1eb-e967-4c64-8b8b-5530e7fb190a" (UID: "6a73c1eb-e967-4c64-8b8b-5530e7fb190a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:01:52.810269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.810245    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4" pod "6a73c1eb-e967-4c64-8b8b-5530e7fb190a" (UID: "6a73c1eb-e967-4c64-8b8b-5530e7fb190a"). InnerVolumeSpecName "kube-api-access-tcqp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:01:52.910231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.910162    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tcqp4\" (UniqueName: \"kubernetes.io/projected/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-kube-api-access-tcqp4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:52.910231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:52.910200    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6a73c1eb-e967-4c64-8b8b-5530e7fb190a-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:01:53.355542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.355511    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:53.355949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.355915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:53.356785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.356765    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:01:53.582136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.582105    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"
Oct 30 22:01:53.582317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.582103    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6" event={"ID":"6a73c1eb-e967-4c64-8b8b-5530e7fb190a","Type":"ContainerDied","Data":"ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5"}
Oct 30 22:01:53.582317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:53.582229    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffac5b6eccce144ac8c4dd087f761bae750e573df69b8e557b5af84fb0fe8ea5"
Oct 30 22:01:58.060411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.060369    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"]
Oct 30 22:01:58.062222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.062191    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6a73c1eb-e967-4c64-8b8b-5530e7fb190a" containerName="revoke-keys"
Oct 30 22:01:58.062407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.062393    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6a73c1eb-e967-4c64-8b8b-5530e7fb190a" containerName="revoke-keys"
Oct 30 22:01:58.062934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.062911    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6a73c1eb-e967-4c64-8b8b-5530e7fb190a" containerName="revoke-keys"
Oct 30 22:01:58.068363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.068341    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.116773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.116741    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"]
Oct 30 22:01:58.162082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.162050    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:01:58.162251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.162188    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:01:58.186677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.186630    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.186885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.186770    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhjvq\" (UniqueName: \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.186885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.186857    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.288732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.288688    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.288942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.288788    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhjvq\" (UniqueName: \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.288942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.288891    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.289564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.289534    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.291394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.291370    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.352783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.352748    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhjvq\" (UniqueName: \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\") pod \"maas-api-revoke-sub-172c9155-cpx7m\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.382078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.382040    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:01:58.758110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:58.758081    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"]
Oct 30 22:01:58.759879 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:01:58.759844    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43b43c12_d6dc_4462_8590_4b019c759962.slice/crio-02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523 WatchSource:0}: Error finding container 02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523: Status 404 returned error can't find the container with id 02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523
Oct 30 22:01:59.625483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.625446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m" event={"ID":"43b43c12-d6dc-4462-8590-4b019c759962","Type":"ContainerStarted","Data":"1eed5e2dac265aabf51685e9ac1209d803f4001ee19c985e20643ff7fc2455dc"}
Oct 30 22:01:59.625956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.625499    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m" event={"ID":"43b43c12-d6dc-4462-8590-4b019c759962","Type":"ContainerStarted","Data":"02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523"}
Oct 30 22:01:59.748693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.748640    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m" podStartSLOduration=1.7485697629999999 podStartE2EDuration="1.748569763s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:01:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:01:59.74370509 +0000 UTC m=+1442.917029510" watchObservedRunningTime="2026-10-30 22:01:59.748569763 +0000 UTC m=+1442.921894190"
Oct 30 22:01:59.809725 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.809684    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"]
Oct 30 22:01:59.813952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.813928    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:01:59.868943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.868912    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"]
Oct 30 22:01:59.909875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.909766    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:01:59.910063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.909908    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:01:59.910126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:01:59.910080    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b6sz\" (UniqueName: \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.011823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.011781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.012005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.011845    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.012005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.011963    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b6sz\" (UniqueName: \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.012572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.012541    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.014244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.014223    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.074176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.074144    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b6sz\" (UniqueName: \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\") pod \"maas-api-revoke-sub-574de1fd-vslps\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.129577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.129544    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:00.529566 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:02:00.529530    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fcc13a4_836c_4024_b2f6_97bc1862b1e1.slice/crio-9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1 WatchSource:0}: Error finding container 9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1: Status 404 returned error can't find the container with id 9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1
Oct 30 22:02:00.532939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.532915    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"]
Oct 30 22:02:00.634194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.634158    4882 generic.go:356] "Generic (PLEG): container finished" podID="43b43c12-d6dc-4462-8590-4b019c759962" containerID="1eed5e2dac265aabf51685e9ac1209d803f4001ee19c985e20643ff7fc2455dc" exitCode=0
Oct 30 22:02:00.634673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.634237    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m" event={"ID":"43b43c12-d6dc-4462-8590-4b019c759962","Type":"ContainerDied","Data":"1eed5e2dac265aabf51685e9ac1209d803f4001ee19c985e20643ff7fc2455dc"}
Oct 30 22:02:00.635766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:00.635737    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps" event={"ID":"5fcc13a4-836c-4024-b2f6-97bc1862b1e1","Type":"ContainerStarted","Data":"9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1"}
Oct 30 22:02:01.645219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.645135    4882 generic.go:356] "Generic (PLEG): container finished" podID="5fcc13a4-836c-4024-b2f6-97bc1862b1e1" containerID="fa980fdcaf7a352e1309d4ae75895c4e5ba886108682f6599b8816f9954971ef" exitCode=0
Oct 30 22:02:01.645219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.645186    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps" event={"ID":"5fcc13a4-836c-4024-b2f6-97bc1862b1e1","Type":"ContainerDied","Data":"fa980fdcaf7a352e1309d4ae75895c4e5ba886108682f6599b8816f9954971ef"}
Oct 30 22:02:01.829880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.829833    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:02:01.939023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.938915    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\") pod \"43b43c12-d6dc-4462-8590-4b019c759962\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") "
Oct 30 22:02:01.939023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.938985    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\" (UniqueName: \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\") pod \"43b43c12-d6dc-4462-8590-4b019c759962\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") "
Oct 30 22:02:01.939274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.939094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\") pod \"43b43c12-d6dc-4462-8590-4b019c759962\" (UID: \"43b43c12-d6dc-4462-8590-4b019c759962\") "
Oct 30 22:02:01.939784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.939754    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca" pod "43b43c12-d6dc-4462-8590-4b019c759962" (UID: "43b43c12-d6dc-4462-8590-4b019c759962"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:01.941792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.941758    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca" pod "43b43c12-d6dc-4462-8590-4b019c759962" (UID: "43b43c12-d6dc-4462-8590-4b019c759962"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:01.942023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:01.941971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq" pod "43b43c12-d6dc-4462-8590-4b019c759962" (UID: "43b43c12-d6dc-4462-8590-4b019c759962"). InnerVolumeSpecName "kube-api-access-fhjvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:02.044385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.041298    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:02.044385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.041341    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhjvq\" (UniqueName: \"kubernetes.io/projected/43b43c12-d6dc-4462-8590-4b019c759962-kube-api-access-fhjvq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:02.044385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.041358    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43b43c12-d6dc-4462-8590-4b019c759962-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:02.655711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.655678    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"
Oct 30 22:02:02.656163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.655717    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m" event={"ID":"43b43c12-d6dc-4462-8590-4b019c759962","Type":"ContainerDied","Data":"02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523"}
Oct 30 22:02:02.656163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.655778    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02429563fb8df9cbcfe071f21a244e1b4297d5bec07ecb84f7ac8a1f0103a523"
Oct 30 22:02:02.822271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.822242    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:02.954349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.954250    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\") pod \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") "
Oct 30 22:02:02.954521 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.954456    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\" (UniqueName: \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\") pod \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") "
Oct 30 22:02:02.954521 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.954511    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\") pod \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\" (UID: \"5fcc13a4-836c-4024-b2f6-97bc1862b1e1\") "
Oct 30 22:02:02.955028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.954995    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca" pod "5fcc13a4-836c-4024-b2f6-97bc1862b1e1" (UID: "5fcc13a4-836c-4024-b2f6-97bc1862b1e1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:02.955210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.955184    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:02.956994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.956966    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca" pod "5fcc13a4-836c-4024-b2f6-97bc1862b1e1" (UID: "5fcc13a4-836c-4024-b2f6-97bc1862b1e1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:02.959493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:02.957166    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz" pod "5fcc13a4-836c-4024-b2f6-97bc1862b1e1" (UID: "5fcc13a4-836c-4024-b2f6-97bc1862b1e1"). InnerVolumeSpecName "kube-api-access-4b6sz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:03.056724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:03.056687    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b6sz\" (UniqueName: \"kubernetes.io/projected/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-kube-api-access-4b6sz\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:03.056724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:03.056724    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fcc13a4-836c-4024-b2f6-97bc1862b1e1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:03.665515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:03.665486    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"
Oct 30 22:02:03.665515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:03.665508    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps" event={"ID":"5fcc13a4-836c-4024-b2f6-97bc1862b1e1","Type":"ContainerDied","Data":"9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1"}
Oct 30 22:02:03.665995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:03.665550    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc11a658000c9273685eea45161bd6647df6f86b2cfb3c35e19afb48cb973c1"
Oct 30 22:02:10.083550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.083471    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:02:10.084055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084041    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5fcc13a4-836c-4024-b2f6-97bc1862b1e1" containerName="revoke-keys"
Oct 30 22:02:10.084099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084056    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fcc13a4-836c-4024-b2f6-97bc1862b1e1" containerName="revoke-keys"
Oct 30 22:02:10.084099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084080    4882 cpu_manager.go:414] "containerMap: removing container" podUID="43b43c12-d6dc-4462-8590-4b019c759962" containerName="revoke-keys"
Oct 30 22:02:10.084099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084087    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43b43c12-d6dc-4462-8590-4b019c759962" containerName="revoke-keys"
Oct 30 22:02:10.084214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084204    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fcc13a4-836c-4024-b2f6-97bc1862b1e1" containerName="revoke-keys"
Oct 30 22:02:10.084248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.084220    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43b43c12-d6dc-4462-8590-4b019c759962" containerName="revoke-keys"
Oct 30 22:02:10.087898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.087881    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.168714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.168666    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:02:10.195404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.195373    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-695f5c-dockercfg-7bnm8\""
Oct 30 22:02:10.195812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.195738    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-695f5c\""
Oct 30 22:02:10.269561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.269405    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flrn4\" (UniqueName: \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.269561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.269527    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.371318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.371224    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flrn4\" (UniqueName: \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.371494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.371329    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.372085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.372060    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.433336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.433299    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flrn4\" (UniqueName: \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\") pod \"payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:10.701688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:10.701568    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:11.077024 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:02:11.076973    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c19d747_4968_4dba_89da_4c2295833576.slice/crio-e9ee380476c0dc49172428a482eb1582e30466882707160836961a8ab1bf2849 WatchSource:0}: Error finding container e9ee380476c0dc49172428a482eb1582e30466882707160836961a8ab1bf2849: Status 404 returned error can't find the container with id e9ee380476c0dc49172428a482eb1582e30466882707160836961a8ab1bf2849
Oct 30 22:02:11.081108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:11.081073    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:02:11.726632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:11.726577    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" event={"ID":"9c19d747-4968-4dba-89da-4c2295833576","Type":"ContainerStarted","Data":"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"}
Oct 30 22:02:11.729178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:11.726655    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" event={"ID":"9c19d747-4968-4dba-89da-4c2295833576","Type":"ContainerStarted","Data":"e9ee380476c0dc49172428a482eb1582e30466882707160836961a8ab1bf2849"}
Oct 30 22:02:11.729178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:11.726684    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:11.851026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:11.850968    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" podStartSLOduration=1.850951842 podStartE2EDuration="1.850951842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:11.847351872 +0000 UTC m=+1455.020676311" watchObservedRunningTime="2026-10-30 22:02:11.850951842 +0000 UTC m=+1455.024276292"
Oct 30 22:02:18.480424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.480385    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:02:18.485625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.485581    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.541719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.541682    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:02:18.578760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.578718    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.578950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.578779    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.578950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.578803    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nqs7\" (UniqueName: \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.579039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.578966    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.579039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.579011    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.579126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.579093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.586497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.586463    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4c260505\"/\"kube-root-ca.crt\""
Oct 30 22:02:18.586497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.586468    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-4c260505\"/\"openshift-service-ca.crt\""
Oct 30 22:02:18.586860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.586845    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4c260505\"/\"e2e-lifecycle-model-4c260505-kserve-self-signed-certs\""
Oct 30 22:02:18.590255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.590233    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-4c260505\"/\"default-dockercfg-29grg\""
Oct 30 22:02:18.680543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680499    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.680797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680580    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.680797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680639    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nqs7\" (UniqueName: \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.680797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680722    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.680797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680783    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.681115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.680857    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.681115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.681055    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.681279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.681255    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.681412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.681307    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.692634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.688198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.692634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.688633    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.761711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.753792    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nqs7\" (UniqueName: \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\") pod \"e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") " pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:18.808218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:18.808181    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:19.249574 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:02:19.249538    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod903e1a7f_cf27_49c0_97cc_b6d0ab6229e8.slice/crio-295399f7d93491a4ea96b667fa587c7ad64a6e2368f12341bd6ad5e1359f5a33 WatchSource:0}: Error finding container 295399f7d93491a4ea96b667fa587c7ad64a6e2368f12341bd6ad5e1359f5a33: Status 404 returned error can't find the container with id 295399f7d93491a4ea96b667fa587c7ad64a6e2368f12341bd6ad5e1359f5a33
Oct 30 22:02:19.253930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:19.253833    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:02:19.823921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:19.823884    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" event={"ID":"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8","Type":"ContainerStarted","Data":"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"}
Oct 30 22:02:19.823921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:19.823929    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" event={"ID":"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8","Type":"ContainerStarted","Data":"295399f7d93491a4ea96b667fa587c7ad64a6e2368f12341bd6ad5e1359f5a33"}
Oct 30 22:02:19.946821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:19.946758    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" podStartSLOduration=1.9467431830000002 podStartE2EDuration="1.946743183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:19.945106048 +0000 UTC m=+1463.118430478" watchObservedRunningTime="2026-10-30 22:02:19.946743183 +0000 UTC m=+1463.120067663"
Oct 30 22:02:20.808568 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:02:20.808517    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:02:20.812199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:20.812171    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:21.979468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:21.979426    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:02:22.214101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.214066    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"]
Oct 30 22:02:22.218770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.218746    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.272035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.271948    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:02:22.272035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.271977    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:02:22.272035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.271953    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"]
Oct 30 22:02:22.329347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.329302    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.329542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.329523    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.329615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.329600    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzxkj\" (UniqueName: \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.431469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.431430    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.431713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.431624    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.431803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.431779    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzxkj\" (UniqueName: \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.432191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.432168    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.434145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.434125    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.501094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.501056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzxkj\" (UniqueName: \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\") pod \"maas-api-revoke-sub-5b80383f-6qvb7\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.533307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.533226    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:22.737448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.737412    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:02:22.965636 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:02:22.962982    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60658c6f_1783_4494_9685_0c73240f6ffc.slice/crio-aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7 WatchSource:0}: Error finding container aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7: Status 404 returned error can't find the container with id aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7
Oct 30 22:02:22.966471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:22.966447    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"]
Oct 30 22:02:23.940598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:23.940539    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7" event={"ID":"60658c6f-1783-4494-9685-0c73240f6ffc","Type":"ContainerStarted","Data":"548ac3f865e6a3abc3fccb7d4d803ced89c1fed9bfc82488cf604ac98f623eb8"}
Oct 30 22:02:23.941216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:23.941189    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7" event={"ID":"60658c6f-1783-4494-9685-0c73240f6ffc","Type":"ContainerStarted","Data":"aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7"}
Oct 30 22:02:25.189612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.181430    4882 generic.go:356] "Generic (PLEG): container finished" podID="60658c6f-1783-4494-9685-0c73240f6ffc" containerID="548ac3f865e6a3abc3fccb7d4d803ced89c1fed9bfc82488cf604ac98f623eb8" exitCode=0
Oct 30 22:02:25.189612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.181748    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7" event={"ID":"60658c6f-1783-4494-9685-0c73240f6ffc","Type":"ContainerDied","Data":"548ac3f865e6a3abc3fccb7d4d803ced89c1fed9bfc82488cf604ac98f623eb8"}
Oct 30 22:02:25.524667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.518916    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:02:25.606159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.606127    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:02:25.606614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.606581    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:25.670541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.669724    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-s4ndm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:25.670541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.669890    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:25.718754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.718710    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-4kb4v container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:02:25.732647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.719037    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:02:25.752606 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.720751    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:25.757084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.753006    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgv88\" (UniqueName: \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:25.757717 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.728860    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-2d2f63\""
Oct 30 22:02:25.757903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.728791    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-2d2f63-dockercfg-bbz47\""
Oct 30 22:02:25.906715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.903079    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:25.906715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.903213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgv88\" (UniqueName: \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:25.906715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:25.904814    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:26.112661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.110665    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgv88\" (UniqueName: \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\") pod \"payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") " pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:26.297106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.297068    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:26.471084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.471011    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:26.531330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.531293    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\") pod \"60658c6f-1783-4494-9685-0c73240f6ffc\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") "
Oct 30 22:02:26.531537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.531336    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\") pod \"60658c6f-1783-4494-9685-0c73240f6ffc\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") "
Oct 30 22:02:26.531537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.531442    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\" (UniqueName: \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\") pod \"60658c6f-1783-4494-9685-0c73240f6ffc\" (UID: \"60658c6f-1783-4494-9685-0c73240f6ffc\") "
Oct 30 22:02:26.531700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.531680    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca" pod "60658c6f-1783-4494-9685-0c73240f6ffc" (UID: "60658c6f-1783-4494-9685-0c73240f6ffc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:02:26.532021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.531995    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:26.533574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.533551    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca" pod "60658c6f-1783-4494-9685-0c73240f6ffc" (UID: "60658c6f-1783-4494-9685-0c73240f6ffc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:02:26.533714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.533689    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj" pod "60658c6f-1783-4494-9685-0c73240f6ffc" (UID: "60658c6f-1783-4494-9685-0c73240f6ffc"). InnerVolumeSpecName "kube-api-access-dzxkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:02:26.633331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.633297    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzxkj\" (UniqueName: \"kubernetes.io/projected/60658c6f-1783-4494-9685-0c73240f6ffc-kube-api-access-dzxkj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:26.633331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.633334    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/60658c6f-1783-4494-9685-0c73240f6ffc-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:02:26.726501 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:02:26.726464    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ee9bc10_4b4f_45f9_b693_f5519f5f39d5.slice/crio-430778caf4b2b3b476d545d4673940c5253d8bd39ea45dbe0118dcdc3b21ef4f WatchSource:0}: Error finding container 430778caf4b2b3b476d545d4673940c5253d8bd39ea45dbe0118dcdc3b21ef4f: Status 404 returned error can't find the container with id 430778caf4b2b3b476d545d4673940c5253d8bd39ea45dbe0118dcdc3b21ef4f
Oct 30 22:02:26.730621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:26.730580    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:02:27.206809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.206762    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" event={"ID":"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5","Type":"ContainerStarted","Data":"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"}
Oct 30 22:02:27.206995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.206840    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" event={"ID":"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5","Type":"ContainerStarted","Data":"430778caf4b2b3b476d545d4673940c5253d8bd39ea45dbe0118dcdc3b21ef4f"}
Oct 30 22:02:27.206995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.206873    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:27.208242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.208222    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"
Oct 30 22:02:27.208371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.208220    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7" event={"ID":"60658c6f-1783-4494-9685-0c73240f6ffc","Type":"ContainerDied","Data":"aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7"}
Oct 30 22:02:27.208371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.208336    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa5e05a5501f68add5fb4721d5c1a10e7b36abb7f36bb98db533dc08c5dfddb7"
Oct 30 22:02:27.326473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:27.326409    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" podStartSLOduration=2.326392771 podStartE2EDuration="2.326392771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:02:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:02:27.323177663 +0000 UTC m=+1470.496502089" watchObservedRunningTime="2026-10-30 22:02:27.326392771 +0000 UTC m=+1470.499717227"
Oct 30 22:02:28.812046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:28.812000    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:28.828050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:28.828019    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:29.234671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:29.234644    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:02:38.218511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:38.218474    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:02:50.809730 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:02:50.809689    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:02:57.122713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:57.122679    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:02:57.129825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:02:57.129801    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:03:00.053562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.053528    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"]
Oct 30 22:03:00.090034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.090001    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"]
Oct 30 22:03:00.103503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.103469    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9e25d20d-7r5nb"]
Oct 30 22:03:00.107713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.107684    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-341eb523-pjjtv"]
Oct 30 22:03:00.949187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.949135    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="818304b9-ec45-4dab-99cb-3b9b9df2e62b" path="/var/lib/kubelet/pods/818304b9-ec45-4dab-99cb-3b9b9df2e62b/volumes"
Oct 30 22:03:00.949900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:00.949874    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0e060a-39a4-4b3e-939d-555a1d40122e" path="/var/lib/kubelet/pods/da0e060a-39a4-4b3e-939d-555a1d40122e/volumes"
Oct 30 22:03:05.474813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:05.474773    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:03:17.061504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:17.061462    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"]
Oct 30 22:03:17.067685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:17.067652    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"]
Oct 30 22:03:17.111453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:17.111410    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-41fff10f-qbhnf"]
Oct 30 22:03:17.116124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:17.116091    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c7365-ntpd4"]
Oct 30 22:03:18.944144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:18.944108    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79d69ded-65ac-4a92-aceb-7f12baac7c0d" path="/var/lib/kubelet/pods/79d69ded-65ac-4a92-aceb-7f12baac7c0d/volumes"
Oct 30 22:03:18.944645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:18.944629    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cdc6bd8a-7ef2-4373-900b-f4b5125f99aa" path="/var/lib/kubelet/pods/cdc6bd8a-7ef2-4373-900b-f4b5125f99aa/volumes"
Oct 30 22:03:20.333010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.332975    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:03:20.333637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.333620    4882 cpu_manager.go:414] "containerMap: removing container" podUID="60658c6f-1783-4494-9685-0c73240f6ffc" containerName="revoke-keys"
Oct 30 22:03:20.333637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.333638    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60658c6f-1783-4494-9685-0c73240f6ffc" containerName="revoke-keys"
Oct 30 22:03:20.333864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.333848    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="60658c6f-1783-4494-9685-0c73240f6ffc" containerName="revoke-keys"
Oct 30 22:03:20.338398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.338379    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.392376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.392343    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9k87\" (UniqueName: \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.392605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.392407    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.395269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.395237    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:03:20.410212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.410180    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f9b7e7\""
Oct 30 22:03:20.439825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.439795    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f9b7e7-dockercfg-q42zj\""
Oct 30 22:03:20.488286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.488247    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:03:20.493382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.493355    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.494004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.493979    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9k87\" (UniqueName: \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.494113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.494053    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.494928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.494903    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.606661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.603106    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:03:20.645098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.645066    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9k87\" (UniqueName: \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\") pod \"payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") " pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.668624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.668543    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmb66\" (UniqueName: \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.668834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.668630    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.669121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.669100    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:20.769927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.769893    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmb66\" (UniqueName: \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.770098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.769935    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.770764 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.770739    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:20.818008 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:20.817962    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:03:20.831780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:20.831747    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmb66\" (UniqueName: \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\") pod \"payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:21.058247 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:21.058179    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8b17c3_0612_46e1_886e_ccbd1e7e05c7.slice/crio-f597085e19bdd4986b0efa08ae17365ff7ccb9cfa39cd85f28f62ca3e6feba35 WatchSource:0}: Error finding container f597085e19bdd4986b0efa08ae17365ff7ccb9cfa39cd85f28f62ca3e6feba35: Status 404 returned error can't find the container with id f597085e19bdd4986b0efa08ae17365ff7ccb9cfa39cd85f28f62ca3e6feba35
Oct 30 22:03:21.062042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.062015    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:03:21.108251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.108221    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:21.487463 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:21.486802    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod094e077c_f7ee_4f22_ab3f_6125ca7fb61f.slice/crio-9318b1020dec03833efc984f59b5a5a37a526c78e19c614825f79e6fa875f516 WatchSource:0}: Error finding container 9318b1020dec03833efc984f59b5a5a37a526c78e19c614825f79e6fa875f516: Status 404 returned error can't find the container with id 9318b1020dec03833efc984f59b5a5a37a526c78e19c614825f79e6fa875f516
Oct 30 22:03:21.491071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.491041    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:03:21.639368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.639331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" event={"ID":"094e077c-f7ee-4f22-ab3f-6125ca7fb61f","Type":"ContainerStarted","Data":"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"}
Oct 30 22:03:21.639550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.639393    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" event={"ID":"094e077c-f7ee-4f22-ab3f-6125ca7fb61f","Type":"ContainerStarted","Data":"9318b1020dec03833efc984f59b5a5a37a526c78e19c614825f79e6fa875f516"}
Oct 30 22:03:21.639550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.639422    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:21.641332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.641299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" event={"ID":"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7","Type":"ContainerStarted","Data":"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"}
Oct 30 22:03:21.641492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.641358    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:21.641492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.641380    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" event={"ID":"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7","Type":"ContainerStarted","Data":"f597085e19bdd4986b0efa08ae17365ff7ccb9cfa39cd85f28f62ca3e6feba35"}
Oct 30 22:03:21.760615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.760484    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" podStartSLOduration=1.760468801 podStartE2EDuration="1.760468801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:21.756129615 +0000 UTC m=+1524.929454045" watchObservedRunningTime="2026-10-30 22:03:21.760468801 +0000 UTC m=+1524.933793227"
Oct 30 22:03:21.880242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:21.880177    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" podStartSLOduration=1.88016012 podStartE2EDuration="1.88016012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:21.875376917 +0000 UTC m=+1525.048701344" watchObservedRunningTime="2026-10-30 22:03:21.88016012 +0000 UTC m=+1525.053484548"
Oct 30 22:03:32.651552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:32.651519    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:03:32.652057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:32.651606    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:03:34.056924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.056877    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"]
Oct 30 22:03:34.107948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.107908    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"]
Oct 30 22:03:34.112160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.112127    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d0ae0b47-k4z56"]
Oct 30 22:03:34.116774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.116740    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e38c6beb-kqc29"]
Oct 30 22:03:34.944814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.944778    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="066a1cd2-b49e-40a3-84c1-b5ac0b9b739f" path="/var/lib/kubelet/pods/066a1cd2-b49e-40a3-84c1-b5ac0b9b739f/volumes"
Oct 30 22:03:34.945364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:34.945342    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a2b8645-1bc6-4bf3-b016-008f09b2dc8f" path="/var/lib/kubelet/pods/2a2b8645-1bc6-4bf3-b016-008f09b2dc8f/volumes"
Oct 30 22:03:43.452799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.452565    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:03:43.458971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.458940    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.513638 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.513598    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:03:43.519184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.519132    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.519382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.519335    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-htd2v\" (UniqueName: \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.555487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.555452    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a5f7b9\""
Oct 30 22:03:43.555683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.555610    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a5f7b9-dockercfg-tkbcq\""
Oct 30 22:03:43.581718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.581679    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:03:43.586212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.586188    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.620679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.620639    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-htd2v\" (UniqueName: \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.620878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.620712    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.620956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.620929    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vdsxn\" (UniqueName: \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.621033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.621019    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.621805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.621782    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.643623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.643575    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:03:43.683147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.683112    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-htd2v\" (UniqueName: \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\") pod \"payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.727762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.724552    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.727762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.724796    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vdsxn\" (UniqueName: \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.727762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.726250    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.782468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.782438    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:43.787546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.787515    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vdsxn\" (UniqueName: \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\") pod \"payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") " pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:43.902554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:43.902082    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:44.185748 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:44.185713    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cdd56c6_601c_4b2a_812f_880713e7622e.slice/crio-bc6d54eb677f8e1d675a9f4ffda96ac5c9ad6d7da3621549374a7746b6539b51 WatchSource:0}: Error finding container bc6d54eb677f8e1d675a9f4ffda96ac5c9ad6d7da3621549374a7746b6539b51: Status 404 returned error can't find the container with id bc6d54eb677f8e1d675a9f4ffda96ac5c9ad6d7da3621549374a7746b6539b51
Oct 30 22:03:44.190176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.190155    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:03:44.322366 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:44.322332    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfe3de7_9fc9_4d09_a4c3_99abaa8b6452.slice/crio-778fae5d1345552ac560298b25853f3fcc470a53b274cd8bdb0a741fccffa28b WatchSource:0}: Error finding container 778fae5d1345552ac560298b25853f3fcc470a53b274cd8bdb0a741fccffa28b: Status 404 returned error can't find the container with id 778fae5d1345552ac560298b25853f3fcc470a53b274cd8bdb0a741fccffa28b
Oct 30 22:03:44.326543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.326517    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:03:44.832448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.832406    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" event={"ID":"9cdd56c6-601c-4b2a-812f-880713e7622e","Type":"ContainerStarted","Data":"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"}
Oct 30 22:03:44.832949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.832468    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" event={"ID":"9cdd56c6-601c-4b2a-812f-880713e7622e","Type":"ContainerStarted","Data":"bc6d54eb677f8e1d675a9f4ffda96ac5c9ad6d7da3621549374a7746b6539b51"}
Oct 30 22:03:44.832949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.832497    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:44.834321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.834297    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" event={"ID":"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452","Type":"ContainerStarted","Data":"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"}
Oct 30 22:03:44.834415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.834342    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" event={"ID":"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452","Type":"ContainerStarted","Data":"778fae5d1345552ac560298b25853f3fcc470a53b274cd8bdb0a741fccffa28b"}
Oct 30 22:03:44.834415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.834375    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:44.955438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:44.955379    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" podStartSLOduration=1.9553626830000002 podStartE2EDuration="1.955362683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:44.95093898 +0000 UTC m=+1548.124263407" watchObservedRunningTime="2026-10-30 22:03:44.955362683 +0000 UTC m=+1548.128687110"
Oct 30 22:03:45.071987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:45.071941    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"]
Oct 30 22:03:45.077253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:45.077223    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd157ae1-4tmxn"]
Oct 30 22:03:45.078224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:45.078180    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" podStartSLOduration=2.078168109 podStartE2EDuration="2.078168109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:45.074616846 +0000 UTC m=+1548.247941263" watchObservedRunningTime="2026-10-30 22:03:45.078168109 +0000 UTC m=+1548.251492536"
Oct 30 22:03:45.442097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:45.442048    4882 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal" podUID="845d50ae5087e4cd62e2c4035d1ecf3f" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:03:46.130992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.130959    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-85f34d"]
Oct 30 22:03:46.135663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.135638    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:46.177896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.177857    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2p6d\" (UniqueName: \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\") pod \"test-curl-7652-85f34d\" (UID: \"06fe856f-c54f-4f73-a285-fb634cdc5a30\") " pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:46.187509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.187474    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-85f34d"]
Oct 30 22:03:46.188967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.188931    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jppg\""
Oct 30 22:03:46.279537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.279504    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2p6d\" (UniqueName: \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\") pod \"test-curl-7652-85f34d\" (UID: \"06fe856f-c54f-4f73-a285-fb634cdc5a30\") " pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:46.341954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.341914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2p6d\" (UniqueName: \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\") pod \"test-curl-7652-85f34d\" (UID: \"06fe856f-c54f-4f73-a285-fb634cdc5a30\") " pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:46.450448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.450358    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:46.833655 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:46.833623    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod06fe856f_c54f_4f73_a285_fb634cdc5a30.slice/crio-9ed7dea96dc144eb41a2691fcfcf2600cae7415fcf00161aeb9fc1dd005c32a6 WatchSource:0}: Error finding container 9ed7dea96dc144eb41a2691fcfcf2600cae7415fcf00161aeb9fc1dd005c32a6: Status 404 returned error can't find the container with id 9ed7dea96dc144eb41a2691fcfcf2600cae7415fcf00161aeb9fc1dd005c32a6
Oct 30 22:03:46.836413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.836391    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:03:46.838347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.838325    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-85f34d"]
Oct 30 22:03:46.853485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.853452    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-85f34d" event={"ID":"06fe856f-c54f-4f73-a285-fb634cdc5a30","Type":"ContainerStarted","Data":"9ed7dea96dc144eb41a2691fcfcf2600cae7415fcf00161aeb9fc1dd005c32a6"}
Oct 30 22:03:46.944396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:46.944362    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d45007a3-1435-458c-b3b3-288f2345b939" path="/var/lib/kubelet/pods/d45007a3-1435-458c-b3b3-288f2345b939/volumes"
Oct 30 22:03:48.868396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:48.868353    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-85f34d" event={"ID":"06fe856f-c54f-4f73-a285-fb634cdc5a30","Type":"ContainerStarted","Data":"58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277"}
Oct 30 22:03:48.987435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:48.987379    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7652-85f34d" podStartSLOduration=1.626509901 podStartE2EDuration="2.987364717s" totalImagesPullingTime="1.360854816s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:48.982523553 +0000 UTC m=+1552.155847981" watchObservedRunningTime="2026-10-30 22:03:48.987364717 +0000 UTC m=+1552.160689143"
Oct 30 22:03:50.548383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:50.548347    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-85f34d"]
Oct 30 22:03:50.555205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:50.555167    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-85f34d"]
Oct 30 22:03:50.809802 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:50.809705    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:03:50.882191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:50.882151    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7652-85f34d" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" containerName="test-curl-7652-85f34d" containerID="cri-o://58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277" gracePeriod=2
Oct 30 22:03:50.933894 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:50.933853    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-85f34d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:51.527715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.527675    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"]
Oct 30 22:03:51.528320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.528306    4882 cpu_manager.go:414] "containerMap: removing container" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" containerName="test-curl-7652-85f34d"
Oct 30 22:03:51.528366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.528321    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" containerName="test-curl-7652-85f34d"
Oct 30 22:03:51.528448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.528439    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" containerName="test-curl-7652-85f34d"
Oct 30 22:03:51.533027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.533008    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.587302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.587271    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"]
Oct 30 22:03:51.632657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.632621    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:03:51.632843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.632686    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:03:51.661979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.661932    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.662195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.662106    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.662195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.662176    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q7vh2\" (UniqueName: \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.763646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.763601    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.763855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.763733    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.763855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.763774    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q7vh2\" (UniqueName: \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.764312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.764287    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.766357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.766325    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.826848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.826770    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q7vh2\" (UniqueName: \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-d9wml\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:51.847759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:51.847728    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:52.060878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.060836    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"]
Oct 30 22:03:52.066923 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.066887    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-227baa92-ncbjb"]
Oct 30 22:03:52.360710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.359397    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"]
Oct 30 22:03:52.364100 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:52.364062    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode39569f4_c3cf_45e7_8d9f_5bc0af665de8.slice/crio-7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278 WatchSource:0}: Error finding container 7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278: Status 404 returned error can't find the container with id 7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278
Oct 30 22:03:52.906025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.905985    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml" event={"ID":"e39569f4-c3cf-45e7-8d9f-5bc0af665de8","Type":"ContainerStarted","Data":"ed078b730d6dff29a2cecdb2d9986e6f5384d0d8cf35a9eebea1dfdb13beedbc"}
Oct 30 22:03:52.906383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.906060    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml" event={"ID":"e39569f4-c3cf-45e7-8d9f-5bc0af665de8","Type":"ContainerStarted","Data":"7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278"}
Oct 30 22:03:52.945314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:52.945275    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d6c496-0bf8-40bb-99ec-e5e028242034" path="/var/lib/kubelet/pods/32d6c496-0bf8-40bb-99ec-e5e028242034/volumes"
Oct 30 22:03:53.029312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.029246    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml" podStartSLOduration=2.029227445 podStartE2EDuration="2.029227445s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:53.022389684 +0000 UTC m=+1556.195714112" watchObservedRunningTime="2026-10-30 22:03:53.029227445 +0000 UTC m=+1556.202551875"
Oct 30 22:03:53.067508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.067466    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"]
Oct 30 22:03:53.072927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.072895    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e04c2c1-q99fd"]
Oct 30 22:03:53.152442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.152363    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:53.231105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.231068    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\" (UniqueName: \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\") pod \"06fe856f-c54f-4f73-a285-fb634cdc5a30\" (UID: \"06fe856f-c54f-4f73-a285-fb634cdc5a30\") "
Oct 30 22:03:53.233314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.233278    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d" pod "06fe856f-c54f-4f73-a285-fb634cdc5a30" (UID: "06fe856f-c54f-4f73-a285-fb634cdc5a30"). InnerVolumeSpecName "kube-api-access-g2p6d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:53.332648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.332614    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2p6d\" (UniqueName: \"kubernetes.io/projected/06fe856f-c54f-4f73-a285-fb634cdc5a30-kube-api-access-g2p6d\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:53.916278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.916226    4882 generic.go:356] "Generic (PLEG): container finished" podID="06fe856f-c54f-4f73-a285-fb634cdc5a30" containerID="58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277" exitCode=137
Oct 30 22:03:53.916778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.916282    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:53.916778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.916445    4882 scope.go:122] "RemoveContainer" containerID="58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277"
Oct 30 22:03:53.919864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.919833    4882 generic.go:356] "Generic (PLEG): container finished" podID="e39569f4-c3cf-45e7-8d9f-5bc0af665de8" containerID="ed078b730d6dff29a2cecdb2d9986e6f5384d0d8cf35a9eebea1dfdb13beedbc" exitCode=0
Oct 30 22:03:53.920047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.920030    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml" event={"ID":"e39569f4-c3cf-45e7-8d9f-5bc0af665de8","Type":"ContainerDied","Data":"ed078b730d6dff29a2cecdb2d9986e6f5384d0d8cf35a9eebea1dfdb13beedbc"}
Oct 30 22:03:53.931708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.931689    4882 scope.go:122] "RemoveContainer" containerID="58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277"
Oct 30 22:03:53.932035 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:53.932017    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277\": container with ID starting with 58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277 not found: ID does not exist" containerID="58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277"
Oct 30 22:03:53.932090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:53.932044    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277"} err="failed to get container status \"58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277\": rpc error: code = NotFound desc = could not find container \"58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277\": container with ID starting with 58b8d7daf9c293ebe1225d966898e6ca0f3d6f247719ff3511acae01d4a85277 not found: ID does not exist"
Oct 30 22:03:54.088978 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:54.088935    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-85f34d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:54.624811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.624780    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:03:54.630856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.630833    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.683189 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:54.683138    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-85f34d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:54.685033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.685003    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-4cb675\""
Oct 30 22:03:54.685203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.685153    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-4cb675-dockercfg-ttm7p\""
Oct 30 22:03:54.688010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.687979    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:03:54.753569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.753523    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.753804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.753675    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7nl2v\" (UniqueName: \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.773941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.773900    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:03:54.778768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.778741    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.841061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.841020    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:03:54.855534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.855486    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p82f7\" (UniqueName: \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.855792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.855617    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.855792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.855681    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7nl2v\" (UniqueName: \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.855792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.855768    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.856766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.856736    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:54.871629 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:03:54.864099    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-85f34d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" pod="opendatahub/test-curl-7652-85f34d"
Oct 30 22:03:54.955125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.954693    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06fe856f-c54f-4f73-a285-fb634cdc5a30" path="/var/lib/kubelet/pods/06fe856f-c54f-4f73-a285-fb634cdc5a30/volumes"
Oct 30 22:03:54.955642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.955418    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acea530e-4302-4832-8b70-f3f3ff846808" path="/var/lib/kubelet/pods/acea530e-4302-4832-8b70-f3f3ff846808/volumes"
Oct 30 22:03:54.957103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.957062    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.957307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.957284    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p82f7\" (UniqueName: \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.958920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.958895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:54.963638 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:54.963611    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7nl2v\" (UniqueName: \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\") pod \"payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:55.035137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.034714    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p82f7\" (UniqueName: \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\") pod \"payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") " pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:55.098034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.097994    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:55.185927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.185351    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:55.245499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.245401    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:55.261088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.261054    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\") pod \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") "
Oct 30 22:03:55.261296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.261172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\") pod \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") "
Oct 30 22:03:55.261296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.261229    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\" (UniqueName: \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\") pod \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\" (UID: \"e39569f4-c3cf-45e7-8d9f-5bc0af665de8\") "
Oct 30 22:03:55.261570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.261539    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca" pod "e39569f4-c3cf-45e7-8d9f-5bc0af665de8" (UID: "e39569f4-c3cf-45e7-8d9f-5bc0af665de8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:03:55.261886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.261862    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:55.263308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.263284    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca" pod "e39569f4-c3cf-45e7-8d9f-5bc0af665de8" (UID: "e39569f4-c3cf-45e7-8d9f-5bc0af665de8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:03:55.263564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.263549    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2" pod "e39569f4-c3cf-45e7-8d9f-5bc0af665de8" (UID: "e39569f4-c3cf-45e7-8d9f-5bc0af665de8"). InnerVolumeSpecName "kube-api-access-q7vh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:03:55.363599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.363540    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:55.363788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.363604    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q7vh2\" (UniqueName: \"kubernetes.io/projected/e39569f4-c3cf-45e7-8d9f-5bc0af665de8-kube-api-access-q7vh2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:03:55.509269 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:55.509230    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4e5d01f_db42_47b9_803d_6240e5a10487.slice/crio-eb9625251d35b80a5d1f8afc9edcc303db0205605a87b5b3cdb53107c0633f29 WatchSource:0}: Error finding container eb9625251d35b80a5d1f8afc9edcc303db0205605a87b5b3cdb53107c0633f29: Status 404 returned error can't find the container with id eb9625251d35b80a5d1f8afc9edcc303db0205605a87b5b3cdb53107c0633f29
Oct 30 22:03:55.514229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.514196    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:03:55.626643 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:03:55.626571    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b30e2a_fba6_44e0_b8a2_4388a75ae502.slice/crio-1e81ba7185c1d971756ee8e20fe1ba186cba5526f07d43edad5345af721daee0 WatchSource:0}: Error finding container 1e81ba7185c1d971756ee8e20fe1ba186cba5526f07d43edad5345af721daee0: Status 404 returned error can't find the container with id 1e81ba7185c1d971756ee8e20fe1ba186cba5526f07d43edad5345af721daee0
Oct 30 22:03:55.631302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.631273    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:03:55.829756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.829663    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:03:55.847182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.847152    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:03:55.847344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.847225    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:03:55.967053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.967014    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" event={"ID":"c4e5d01f-db42-47b9-803d-6240e5a10487","Type":"ContainerStarted","Data":"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"}
Oct 30 22:03:55.967496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.967082    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" event={"ID":"c4e5d01f-db42-47b9-803d-6240e5a10487","Type":"ContainerStarted","Data":"eb9625251d35b80a5d1f8afc9edcc303db0205605a87b5b3cdb53107c0633f29"}
Oct 30 22:03:55.967496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.967115    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:03:55.969115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.969088    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" event={"ID":"55b30e2a-fba6-44e0-b8a2-4388a75ae502","Type":"ContainerStarted","Data":"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"}
Oct 30 22:03:55.969244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.969136    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" event={"ID":"55b30e2a-fba6-44e0-b8a2-4388a75ae502","Type":"ContainerStarted","Data":"1e81ba7185c1d971756ee8e20fe1ba186cba5526f07d43edad5345af721daee0"}
Oct 30 22:03:55.969244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.969190    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:03:55.970527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.970503    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml" event={"ID":"e39569f4-c3cf-45e7-8d9f-5bc0af665de8","Type":"ContainerDied","Data":"7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278"}
Oct 30 22:03:55.970672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.970521    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"
Oct 30 22:03:55.970672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:55.970544    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7809bf3f7e51133153842d5427818b65840177af37b2e4451d75835a23a71278"
Oct 30 22:03:56.267895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:56.267839    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" podStartSLOduration=2.267824738 podStartE2EDuration="2.267824738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.262961881 +0000 UTC m=+1559.436286308" watchObservedRunningTime="2026-10-30 22:03:56.267824738 +0000 UTC m=+1559.441149165"
Oct 30 22:03:56.384222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:56.384154    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" podStartSLOduration=2.384134456 podStartE2EDuration="2.384134456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:03:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:03:56.380886447 +0000 UTC m=+1559.554210875" watchObservedRunningTime="2026-10-30 22:03:56.384134456 +0000 UTC m=+1559.557458883"
Oct 30 22:03:57.544821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:57.544791    4882 scope.go:122] "RemoveContainer" containerID="22e216e6c40adc0c9b51daa762d7bf712dc2f3d8b471749338d811e307244e6d"
Oct 30 22:03:57.557424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:03:57.557394    4882 scope.go:122] "RemoveContainer" containerID="8e0af14b0b29921be3ec444c8aa19077f2a16ada380d95ac6500b66309589b42"
Oct 30 22:04:01.056550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:01.056514    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"]
Oct 30 22:04:01.106740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:01.106693    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"]
Oct 30 22:04:01.111112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:01.111075    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-125d02ff-79szd"]
Oct 30 22:04:01.115085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:01.115060    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-811d30e7-zphn4"]
Oct 30 22:04:02.112923 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.112888    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:04:02.113651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.113633    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e39569f4-c3cf-45e7-8d9f-5bc0af665de8" containerName="revoke-keys"
Oct 30 22:04:02.113651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.113651    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e39569f4-c3cf-45e7-8d9f-5bc0af665de8" containerName="revoke-keys"
Oct 30 22:04:02.113814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.113803    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e39569f4-c3cf-45e7-8d9f-5bc0af665de8" containerName="revoke-keys"
Oct 30 22:04:02.119036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.119017    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.174271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.174233    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:04:02.218548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.218514    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-82b62e\""
Oct 30 22:04:02.218548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.218514    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-82b62e-dockercfg-6m4vj\""
Oct 30 22:04:02.248958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.248921    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:04:02.254536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.254512    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.255902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.255868    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hx87\" (UniqueName: \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.256085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.256062    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.312143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.312097    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:04:02.357378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.357337    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.357545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.357388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skzft\" (UniqueName: \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.357545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.357465    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hx87\" (UniqueName: \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.357710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.357678    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.359525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.358701    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.430394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.430309    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hx87\" (UniqueName: \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\") pod \"payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.434061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.434028    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:02.459945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.459909    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.460083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.459960    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skzft\" (UniqueName: \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.460952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.460924    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.524249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.524213    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skzft\" (UniqueName: \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\") pod \"payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") " pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.570062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.569972    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:02.822558 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:04:02.822521    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16a0c2a8_27de_4e53_bb27_3b91a4906119.slice/crio-bf9ab7452aebf991e4ab508366bb98d1bdd6ab5331b94adf7f0d9c1c9c47bbfb WatchSource:0}: Error finding container bf9ab7452aebf991e4ab508366bb98d1bdd6ab5331b94adf7f0d9c1c9c47bbfb: Status 404 returned error can't find the container with id bf9ab7452aebf991e4ab508366bb98d1bdd6ab5331b94adf7f0d9c1c9c47bbfb
Oct 30 22:04:02.828341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.828306    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:04:02.944160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.944128    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f4410dd-c22c-4cc4-9632-80166b19c500" path="/var/lib/kubelet/pods/1f4410dd-c22c-4cc4-9632-80166b19c500/volumes"
Oct 30 22:04:02.944665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.944650    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="750c3524-e7fe-452d-9f22-b0059092ee36" path="/var/lib/kubelet/pods/750c3524-e7fe-452d-9f22-b0059092ee36/volumes"
Oct 30 22:04:02.972065 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:04:02.972027    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb91bbde5_b20c_4f0a_8ae7_e1360a9b8f37.slice/crio-5ee66be65b1615d73a07981ba812357d7cab49330c9883545890a183ebd13c59 WatchSource:0}: Error finding container 5ee66be65b1615d73a07981ba812357d7cab49330c9883545890a183ebd13c59: Status 404 returned error can't find the container with id 5ee66be65b1615d73a07981ba812357d7cab49330c9883545890a183ebd13c59
Oct 30 22:04:02.976528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:02.976502    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:04:03.025554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:03.025515    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" event={"ID":"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37","Type":"ContainerStarted","Data":"5ee66be65b1615d73a07981ba812357d7cab49330c9883545890a183ebd13c59"}
Oct 30 22:04:03.028599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:03.028549    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" event={"ID":"16a0c2a8-27de-4e53-bb27-3b91a4906119","Type":"ContainerStarted","Data":"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"}
Oct 30 22:04:03.028741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:03.028649    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" event={"ID":"16a0c2a8-27de-4e53-bb27-3b91a4906119","Type":"ContainerStarted","Data":"bf9ab7452aebf991e4ab508366bb98d1bdd6ab5331b94adf7f0d9c1c9c47bbfb"}
Oct 30 22:04:03.028741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:03.028707    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:03.151162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:03.151099    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" podStartSLOduration=1.151080823 podStartE2EDuration="1.151080823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:03.147252988 +0000 UTC m=+1566.320577414" watchObservedRunningTime="2026-10-30 22:04:03.151080823 +0000 UTC m=+1566.324405249"
Oct 30 22:04:04.038763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:04.038727    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" event={"ID":"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37","Type":"ContainerStarted","Data":"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"}
Oct 30 22:04:04.054101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:04.054067    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"]
Oct 30 22:04:04.104799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:04.104764    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-df0bbd04-94mt2"]
Oct 30 22:04:04.169861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:04.169798    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" podStartSLOduration=2.169782187 podStartE2EDuration="2.169782187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:04:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:04:04.167865121 +0000 UTC m=+1567.341189551" watchObservedRunningTime="2026-10-30 22:04:04.169782187 +0000 UTC m=+1567.343106636"
Oct 30 22:04:04.945790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:04.945742    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaa58f37-bfe7-4889-8666-f9c0a89086c6" path="/var/lib/kubelet/pods/eaa58f37-bfe7-4889-8666-f9c0a89086c6/volumes"
Oct 30 22:04:05.046683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:05.046649    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:06.981559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:06.981525    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:04:06.982112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:06.981656    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:04:07.438742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.438703    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"]
Oct 30 22:04:07.443452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.443430    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.496427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.496387    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:04:07.496641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.496577    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:04:07.499275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.499243    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"]
Oct 30 22:04:07.535075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.535039    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.535075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.535089    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng5r8\" (UniqueName: \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.535366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.535204    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.637030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.636980    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.637233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.637042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng5r8\" (UniqueName: \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.637233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.637115    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.637820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.637798    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.639404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.639382    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.699092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.699010    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng5r8\" (UniqueName: \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\") pod \"maas-api-revoke-sub-6804b179-97fjx\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:07.758610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:07.758566    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:08.132561 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:04:08.132521    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62650cb7_cda5_4542_b6c7_30edf4954387.slice/crio-dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51 WatchSource:0}: Error finding container dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51: Status 404 returned error can't find the container with id dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51
Oct 30 22:04:08.135990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:08.135881    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"]
Oct 30 22:04:09.080776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:09.080738    4882 generic.go:356] "Generic (PLEG): container finished" podID="62650cb7-cda5-4542-b6c7-30edf4954387" containerID="d292fdce8c20b2627b6f9a98c0fe856be930d6035ed2b88e05bd3a979d347279" exitCode=0
Oct 30 22:04:09.080925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:09.080785    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx" event={"ID":"62650cb7-cda5-4542-b6c7-30edf4954387","Type":"ContainerDied","Data":"d292fdce8c20b2627b6f9a98c0fe856be930d6035ed2b88e05bd3a979d347279"}
Oct 30 22:04:09.080925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:09.080846    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx" event={"ID":"62650cb7-cda5-4542-b6c7-30edf4954387","Type":"ContainerStarted","Data":"dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51"}
Oct 30 22:04:10.240486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.240454    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:10.272722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.272689    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\" (UniqueName: \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\") pod \"62650cb7-cda5-4542-b6c7-30edf4954387\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") "
Oct 30 22:04:10.272908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.272894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\") pod \"62650cb7-cda5-4542-b6c7-30edf4954387\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") "
Oct 30 22:04:10.272984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.272936    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\") pod \"62650cb7-cda5-4542-b6c7-30edf4954387\" (UID: \"62650cb7-cda5-4542-b6c7-30edf4954387\") "
Oct 30 22:04:10.273380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.273345    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca" pod "62650cb7-cda5-4542-b6c7-30edf4954387" (UID: "62650cb7-cda5-4542-b6c7-30edf4954387"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:04:10.273524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.273504    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.274965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.274938    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca" pod "62650cb7-cda5-4542-b6c7-30edf4954387" (UID: "62650cb7-cda5-4542-b6c7-30edf4954387"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:04:10.275116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.275095    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8" pod "62650cb7-cda5-4542-b6c7-30edf4954387" (UID: "62650cb7-cda5-4542-b6c7-30edf4954387"). InnerVolumeSpecName "kube-api-access-ng5r8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:04:10.375209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.375122    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62650cb7-cda5-4542-b6c7-30edf4954387-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:10.375209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:10.375158    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng5r8\" (UniqueName: \"kubernetes.io/projected/62650cb7-cda5-4542-b6c7-30edf4954387-kube-api-access-ng5r8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:04:11.103182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:11.103145    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx" event={"ID":"62650cb7-cda5-4542-b6c7-30edf4954387","Type":"ContainerDied","Data":"dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51"}
Oct 30 22:04:11.103362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:11.103202    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc88b273c4a9d77c9a9eb095ff8a349e63abcfca398db12b0faee9ecb4a40e51"
Oct 30 22:04:11.103362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:11.103157    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"
Oct 30 22:04:12.057078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:12.057043    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"]
Oct 30 22:04:12.107324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:12.107291    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1fe0330d-dnzs2"]
Oct 30 22:04:12.945278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:12.945230    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="523191c8-0779-48a7-b681-6e7daa54b12c" path="/var/lib/kubelet/pods/523191c8-0779-48a7-b681-6e7daa54b12c/volumes"
Oct 30 22:04:14.042073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:14.042044    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:04:16.055109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:16.055068    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"]
Oct 30 22:04:16.057696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:16.057672    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:04:16.104557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:16.104524    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c48c0b73-fjn2k"]
Oct 30 22:04:16.944314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:16.944282    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f" path="/var/lib/kubelet/pods/39ea5714-7f7e-46d2-9ebf-3d8994ea7a5f/volumes"
Oct 30 22:04:20.810242 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:04:20.810198    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:04:22.055083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:22.055029    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"]
Oct 30 22:04:22.109722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:22.109684    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9f11b7b5-652d4"]
Oct 30 22:04:22.950958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:22.950921    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73a5fc07-1c1d-4e3d-8b90-d98acba23cab" path="/var/lib/kubelet/pods/73a5fc07-1c1d-4e3d-8b90-d98acba23cab/volumes"
Oct 30 22:04:38.056996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:38.056952    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"]
Oct 30 22:04:38.107278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:38.107244    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1591f433-b8s7z"]
Oct 30 22:04:38.944883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:38.944850    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d6889ad-f9db-4c84-929d-d7aec3e005f6" path="/var/lib/kubelet/pods/4d6889ad-f9db-4c84-929d-d7aec3e005f6/volumes"
Oct 30 22:04:42.058705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:42.058668    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"]
Oct 30 22:04:42.108760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:42.108722    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f829486c-s6dq4"]
Oct 30 22:04:42.944270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:42.944238    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="027a9be8-074d-459d-946f-93c060a0636f" path="/var/lib/kubelet/pods/027a9be8-074d-459d-946f-93c060a0636f/volumes"
Oct 30 22:04:44.693427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:44.693378    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:04:49.061225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:49.061190    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"]
Oct 30 22:04:49.111413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:49.111376    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9808ceaf-m2mzx"]
Oct 30 22:04:50.807187 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:04:50.807149    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:04:50.945313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:50.945276    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4229bcaa-d1d8-491c-a588-2d519ead5a08" path="/var/lib/kubelet/pods/4229bcaa-d1d8-491c-a588-2d519ead5a08/volumes"
Oct 30 22:04:57.611135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.611100    4882 scope.go:122] "RemoveContainer" containerID="7e2a14d1035ef3aaab5b63307521fc3161eaeffaf7638616984b941562c61faa"
Oct 30 22:04:57.622095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.622073    4882 scope.go:122] "RemoveContainer" containerID="6701458106faeab96d7386a99d0223af998941d1b44530d18802673cfef9d4a4"
Oct 30 22:04:57.632382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.632362    4882 scope.go:122] "RemoveContainer" containerID="21c7c27ba8cc4b54d3e9bca7c5f95c757836a15870ef0a623b3bbcd041d45817"
Oct 30 22:04:57.641851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.641828    4882 scope.go:122] "RemoveContainer" containerID="1e211c1ce167cfd6209c6493f32afb5569adab4393472ba8db00e234d2d7f1cb"
Oct 30 22:04:57.651212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.651193    4882 scope.go:122] "RemoveContainer" containerID="628995f039ce73e3349f173c67bed560c41f29d265fc5a6bdbc02c97065c5f53"
Oct 30 22:04:57.660364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.660340    4882 scope.go:122] "RemoveContainer" containerID="8c36478cd3ebf120a49508da64f736e92280e5de4874c640233ffaabacafe876"
Oct 30 22:04:57.669394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.669369    4882 scope.go:122] "RemoveContainer" containerID="5ce2302a67737d0021dd6776afb3416cfc3fc1a38fee2f1674ab0e26e46a5d94"
Oct 30 22:04:57.678620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:57.678600    4882 scope.go:122] "RemoveContainer" containerID="fce362c244974a0efe4754641d5d4d572e5716b0be00025c684fddc51667bb2e"
Oct 30 22:04:58.050526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:58.050438    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 22:04:58.050740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:58.050712    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy" containerID="cri-o://489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe" gracePeriod=30
Oct 30 22:04:58.101288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:58.101251    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 22:04:58.101506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:04:58.101475    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy" containerID="cri-o://d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b" gracePeriod=30
Oct 30 22:05:03.400661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.400556    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 22:05:03.422334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.422308    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 22:05:03.502799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.502761    4882 generic.go:356] "Generic (PLEG): container finished" podID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerID="489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe" exitCode=0
Oct 30 22:05:03.502983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.502812    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"
Oct 30 22:05:03.502983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.502845    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" event={"ID":"8b41a63c-ea14-44f0-9ade-811b708624ed","Type":"ContainerDied","Data":"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"}
Oct 30 22:05:03.502983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.502904    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv" event={"ID":"8b41a63c-ea14-44f0-9ade-811b708624ed","Type":"ContainerDied","Data":"c5321ab9ce21fea42b123852f73f5ddf69a7ea4a5b4862f1241386b656d854c1"}
Oct 30 22:05:03.502983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.502928    4882 scope.go:122] "RemoveContainer" containerID="489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"
Oct 30 22:05:03.504610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.504555    4882 generic.go:356] "Generic (PLEG): container finished" podID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerID="d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b" exitCode=0
Oct 30 22:05:03.504738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.504623    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" event={"ID":"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e","Type":"ContainerDied","Data":"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"}
Oct 30 22:05:03.504738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.504672    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8" event={"ID":"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e","Type":"ContainerDied","Data":"45b42e5618b3a2bbcecbae2d6ab63ff6770558f4f4877506dea0a6fdc502c571"}
Oct 30 22:05:03.504738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.504630    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"
Oct 30 22:05:03.512978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.512948    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513012    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513045    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513125    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.513328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513199    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513240    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513303    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.513558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513341    4882 scope.go:122] "RemoveContainer" containerID="489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"
Oct 30 22:05:03.513558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513375    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513403    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513456    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.513558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513495    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.513846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513564    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\") pod \"8b41a63c-ea14-44f0-9ade-811b708624ed\" (UID: \"8b41a63c-ea14-44f0-9ade-811b708624ed\") "
Oct 30 22:05:03.513846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513644    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.513846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513669    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.514060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513869    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.514060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.513895    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.514445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514196    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.514445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.514445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514245    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:03.514728 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:05:03.514566    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe\": container with ID starting with 489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe not found: ID does not exist" containerID="489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"
Oct 30 22:05:03.514728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514647    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe"} err="failed to get container status \"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe\": rpc error: code = NotFound desc = could not find container \"489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe\": container with ID starting with 489fc36be69c3ce9194e52564912a23420598390bc4dcbb123789cfef9b9a4fe not found: ID does not exist"
Oct 30 22:05:03.514728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514675    4882 scope.go:122] "RemoveContainer" containerID="d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514794    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514817    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514835    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514850    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514864    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8b41a63c-ea14-44f0-9ade-811b708624ed-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514879    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.514895    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.515800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.515778    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:03.515910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.515864    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "kube-api-access-dt2wh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:03.516311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.516285    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:03.516402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.516386    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.516643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.516622    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "kube-api-access-48smv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:03.517671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.517648    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo" pod "8b41a63c-ea14-44f0-9ade-811b708624ed" (UID: "8b41a63c-ea14-44f0-9ade-811b708624ed"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:03.525256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.525235    4882 scope.go:122] "RemoveContainer" containerID="d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"
Oct 30 22:05:03.525724 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:05:03.525692    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b\": container with ID starting with d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b not found: ID does not exist" containerID="d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"
Oct 30 22:05:03.525825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.525733    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b"} err="failed to get container status \"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b\": rpc error: code = NotFound desc = could not find container \"d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b\": container with ID starting with d7e811b1d41cfcdf97fdcfc87f628ae456f18ea4ade2c1c45e31e9c0be7ce97b not found: ID does not exist"
Oct 30 22:05:03.616188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.616354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616189    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.616354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.616485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616441    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.616549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616498    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.616549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616515    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.616683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616554    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:03.616683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.616563    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\") pod \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\" (UID: \"a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e\") "
Oct 30 22:05:03.617138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617090    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617116    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617131    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617145    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617160    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-48smv\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-kube-api-access-48smv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617174    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617185    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617197    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8b41a63c-ea14-44f0-9ade-811b708624ed-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.617384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.617209    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt2wh\" (UniqueName: \"kubernetes.io/projected/8b41a63c-ea14-44f0-9ade-811b708624ed-kube-api-access-dt2wh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.619290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.619260    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:03.619403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.619294    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token" pod "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" (UID: "a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:03.718489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.718456    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.718489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.718486    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:03.932737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.932645    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 22:05:03.938724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:03.938692    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-q27sv"]
Oct 30 22:05:04.057882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:04.057852    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 22:05:04.061398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:04.061378    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-94bt8"]
Oct 30 22:05:04.946096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:04.946055    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" path="/var/lib/kubelet/pods/8b41a63c-ea14-44f0-9ade-811b708624ed/volumes"
Oct 30 22:05:04.946774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:04.946753    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" path="/var/lib/kubelet/pods/a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e/volumes"
Oct 30 22:05:13.415776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:13.415692    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"]
Oct 30 22:05:13.419380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:13.419354    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-d9wml"]
Oct 30 22:05:14.944626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:14.944571    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e39569f4-c3cf-45e7-8d9f-5bc0af665de8" path="/var/lib/kubelet/pods/e39569f4-c3cf-45e7-8d9f-5bc0af665de8/volumes"
Oct 30 22:05:15.204078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:15.203993    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:05:19.812362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.812329    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:05:19.812977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.812958    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy"
Oct 30 22:05:19.812977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.812978    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy"
Oct 30 22:05:19.813078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.812993    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy"
Oct 30 22:05:19.813078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.812998    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy"
Oct 30 22:05:19.813078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.813011    4882 cpu_manager.go:414] "containerMap: removing container" podUID="62650cb7-cda5-4542-b6c7-30edf4954387" containerName="revoke-keys"
Oct 30 22:05:19.813078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.813017    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62650cb7-cda5-4542-b6c7-30edf4954387" containerName="revoke-keys"
Oct 30 22:05:19.813194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.813139    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4fd6f23-4b8c-4104-8fe8-20d19f9d7d9e" containerName="istio-proxy"
Oct 30 22:05:19.813194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.813147    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b41a63c-ea14-44f0-9ade-811b708624ed" containerName="istio-proxy"
Oct 30 22:05:19.813194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.813171    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="62650cb7-cda5-4542-b6c7-30edf4954387" containerName="revoke-keys"
Oct 30 22:05:19.816664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.816646    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:19.873047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.873008    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:05:19.933060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.933021    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-97f097\""
Oct 30 22:05:19.933256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.933161    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-97f097-dockercfg-gngd6\""
Oct 30 22:05:19.941034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.941002    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:05:19.947482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.947449    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:19.974350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.974307    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v9p5p\" (UniqueName: \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:19.976283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.974801    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:19.989896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:19.989862    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:05:20.078631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.078563    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwddv\" (UniqueName: \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.078920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.078896    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:20.079032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.079014    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.079155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.079140    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v9p5p\" (UniqueName: \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:20.079910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.079867    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:20.146147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.146074    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v9p5p\" (UniqueName: \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\") pod \"payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:20.180561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.180510    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.180778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.180739    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwddv\" (UniqueName: \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.181245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.181224    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.244054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.244018    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwddv\" (UniqueName: \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\") pod \"payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") " pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.268675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.268640    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:20.431815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.431718    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:20.651352 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:05:20.651303    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e018ba_a061_4838_aa21_ef0c0d1d408d.slice/crio-36a205757c27184ad5a8f170efbb1e954bb762c372eff709ec9ea474f5f29c2e WatchSource:0}: Error finding container 36a205757c27184ad5a8f170efbb1e954bb762c372eff709ec9ea474f5f29c2e: Status 404 returned error can't find the container with id 36a205757c27184ad5a8f170efbb1e954bb762c372eff709ec9ea474f5f29c2e
Oct 30 22:05:20.654973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.654947    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:05:20.675000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.674917    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" event={"ID":"e2e018ba-a061-4838-aa21-ef0c0d1d408d","Type":"ContainerStarted","Data":"36a205757c27184ad5a8f170efbb1e954bb762c372eff709ec9ea474f5f29c2e"}
Oct 30 22:05:20.810522 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:05:20.810489    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:05:20.818134 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:05:20.818099    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2b9d4c76_ccc6_4fcf_ba16_7ed6578c1dc1.slice/crio-9bd5ab276d9c51f4fef4e2c32a35dc0495ce6f09de325ed96fb7b0d15ce2e916 WatchSource:0}: Error finding container 9bd5ab276d9c51f4fef4e2c32a35dc0495ce6f09de325ed96fb7b0d15ce2e916: Status 404 returned error can't find the container with id 9bd5ab276d9c51f4fef4e2c32a35dc0495ce6f09de325ed96fb7b0d15ce2e916
Oct 30 22:05:20.822004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:20.821979    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:05:21.053271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.053169    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"]
Oct 30 22:05:21.056899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.056869    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01ecd803-5x49c"]
Oct 30 22:05:21.684451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.684411    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" event={"ID":"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1","Type":"ContainerStarted","Data":"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"}
Oct 30 22:05:21.684668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.684473    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" event={"ID":"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1","Type":"ContainerStarted","Data":"9bd5ab276d9c51f4fef4e2c32a35dc0495ce6f09de325ed96fb7b0d15ce2e916"}
Oct 30 22:05:21.684668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.684501    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:21.686324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.686214    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" event={"ID":"e2e018ba-a061-4838-aa21-ef0c0d1d408d","Type":"ContainerStarted","Data":"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"}
Oct 30 22:05:21.686324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.686271    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:21.809358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.809303    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" podStartSLOduration=2.809287148 podStartE2EDuration="2.809287148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:21.80656142 +0000 UTC m=+1644.979885849" watchObservedRunningTime="2026-10-30 22:05:21.809287148 +0000 UTC m=+1644.982611626"
Oct 30 22:05:21.930093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:21.930039    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" podStartSLOduration=2.930024424 podStartE2EDuration="2.930024424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:21.927450574 +0000 UTC m=+1645.100775002" watchObservedRunningTime="2026-10-30 22:05:21.930024424 +0000 UTC m=+1645.103348850"
Oct 30 22:05:22.944779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:22.944742    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8213404-5b59-4fb2-b200-274500dd4dc0" path="/var/lib/kubelet/pods/c8213404-5b59-4fb2-b200-274500dd4dc0/volumes"
Oct 30 22:05:30.513300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:30.513261    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 22:05:30.513875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:30.513480    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy" containerID="cri-o://4cdcc51a2a4ee22ce7dbcbf89e847fd92bb24ac602c9792b2f1976dd60af364b" gracePeriod=30
Oct 30 22:05:30.564029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:30.563992    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 22:05:30.564281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:30.564247    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy" containerID="cri-o://35bff7387427211f285ae61853c4702d4cd036d61a832a4bfd99003407db583a" gracePeriod=30
Oct 30 22:05:32.697275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:32.697246    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:05:32.697745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:32.697325    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:05:35.814414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.814360    4882 generic.go:356] "Generic (PLEG): container finished" podID="77d609a1-ec75-4947-ae8a-588901300e7b" containerID="4cdcc51a2a4ee22ce7dbcbf89e847fd92bb24ac602c9792b2f1976dd60af364b" exitCode=0
Oct 30 22:05:35.814926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.814749    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" event={"ID":"77d609a1-ec75-4947-ae8a-588901300e7b","Type":"ContainerDied","Data":"4cdcc51a2a4ee22ce7dbcbf89e847fd92bb24ac602c9792b2f1976dd60af364b"}
Oct 30 22:05:35.817672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.817486    4882 generic.go:356] "Generic (PLEG): container finished" podID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerID="35bff7387427211f285ae61853c4702d4cd036d61a832a4bfd99003407db583a" exitCode=0
Oct 30 22:05:35.817672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.817566    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" event={"ID":"673ea3b3-f9ae-4e71-83b5-eddeaf89299e","Type":"ContainerDied","Data":"35bff7387427211f285ae61853c4702d4cd036d61a832a4bfd99003407db583a"}
Oct 30 22:05:35.856508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.856475    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 22:05:35.880466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:35.880439    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 22:05:36.000900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.000864    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.000965    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001004    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001041    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001090    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001141    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001189    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001218    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001270    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001303    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001321    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001340    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001342    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001398    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001414    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001535    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001565    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001578    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001649    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001691    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\") pod \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\" (UID: \"673ea3b3-f9ae-4e71-83b5-eddeaf89299e\") "
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001726    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001862    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.001924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001893    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.002390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.001942    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\") pod \"77d609a1-ec75-4947-ae8a-588901300e7b\" (UID: \"77d609a1-ec75-4947-ae8a-588901300e7b\") "
Oct 30 22:05:36.002390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002085    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.002390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002280    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002559    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002710    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002736    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002752    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002780    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002795    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002809    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002822    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002836    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.002869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.002851    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.004247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.004247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004224    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:36.004247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004226    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "kube-api-access-djpbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:36.004403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004263    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token" pod "673ea3b3-f9ae-4e71-83b5-eddeaf89299e" (UID: "673ea3b3-f9ae-4e71-83b5-eddeaf89299e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:36.004530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004508    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:36.004875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004857    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:05:36.004935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.004908    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:05:36.005722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.005708    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd" pod "77d609a1-ec75-4947-ae8a-588901300e7b" (UID: "77d609a1-ec75-4947-ae8a-588901300e7b"). InnerVolumeSpecName "kube-api-access-7cfhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:36.104712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104678    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-djpbh\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-kube-api-access-djpbh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104709    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/77d609a1-ec75-4947-ae8a-588901300e7b-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104719    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104728    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104738    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104748    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/673ea3b3-f9ae-4e71-83b5-eddeaf89299e-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104757    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cfhd\" (UniqueName: \"kubernetes.io/projected/77d609a1-ec75-4947-ae8a-588901300e7b-kube-api-access-7cfhd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104765    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/77d609a1-ec75-4947-ae8a-588901300e7b-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.104967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.104774    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/77d609a1-ec75-4947-ae8a-588901300e7b-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:36.828462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.828375    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"
Oct 30 22:05:36.828946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.828667    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q" event={"ID":"77d609a1-ec75-4947-ae8a-588901300e7b","Type":"ContainerDied","Data":"683eae707c85213123f4bc2c5b28a843ac50b0bd5468aab4c2754a734a05c64d"}
Oct 30 22:05:36.828946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.828748    4882 scope.go:122] "RemoveContainer" containerID="4cdcc51a2a4ee22ce7dbcbf89e847fd92bb24ac602c9792b2f1976dd60af364b"
Oct 30 22:05:36.830555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.830527    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"
Oct 30 22:05:36.830804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.830558    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs" event={"ID":"673ea3b3-f9ae-4e71-83b5-eddeaf89299e","Type":"ContainerDied","Data":"370d3a64106e56d68e5b37719e9402fd4470caba7d01a23232115b5118063427"}
Oct 30 22:05:36.843168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:36.843151    4882 scope.go:122] "RemoveContainer" containerID="35bff7387427211f285ae61853c4702d4cd036d61a832a4bfd99003407db583a"
Oct 30 22:05:37.048670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:37.048634    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 22:05:37.099446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:37.099414    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-q4l6q"]
Oct 30 22:05:37.167175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:37.167137    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 22:05:37.174039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:37.174008    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-fmbzs"]
Oct 30 22:05:38.948103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:38.948067    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" path="/var/lib/kubelet/pods/673ea3b3-f9ae-4e71-83b5-eddeaf89299e/volumes"
Oct 30 22:05:38.948553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:38.948537    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" path="/var/lib/kubelet/pods/77d609a1-ec75-4947-ae8a-588901300e7b/volumes"
Oct 30 22:05:39.813159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.813124    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:05:39.813891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.813870    4882 cpu_manager.go:414] "containerMap: removing container" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy"
Oct 30 22:05:39.813891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.813890    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy"
Oct 30 22:05:39.814043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.813908    4882 cpu_manager.go:414] "containerMap: removing container" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy"
Oct 30 22:05:39.814043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.813914    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy"
Oct 30 22:05:39.814109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.814053    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="673ea3b3-f9ae-4e71-83b5-eddeaf89299e" containerName="istio-proxy"
Oct 30 22:05:39.814109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.814064    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="77d609a1-ec75-4947-ae8a-588901300e7b" containerName="istio-proxy"
Oct 30 22:05:39.819671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.819644    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:39.856041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.856006    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q44gl\" (UniqueName: \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:39.856235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.856135    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:39.927419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.927376    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-4c260505\""
Oct 30 22:05:39.927630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.927445    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-4c260505-dockercfg-8n6k2\""
Oct 30 22:05:39.931313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.931287    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:05:39.957917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.957869    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:39.958659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.958134    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q44gl\" (UniqueName: \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:39.958936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:39.958734    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:40.024064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.024025    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q44gl\" (UniqueName: \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\") pod \"payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:40.135513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.135477    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:40.523930 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:05:40.523375    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3030a906_5209_413d_b120_c919c53ab069.slice/crio-4d655d0389af797af13d72c6b3b4b73fcab8913f1f4c27958237831627b96058 WatchSource:0}: Error finding container 4d655d0389af797af13d72c6b3b4b73fcab8913f1f4c27958237831627b96058: Status 404 returned error can't find the container with id 4d655d0389af797af13d72c6b3b4b73fcab8913f1f4c27958237831627b96058
Oct 30 22:05:40.527007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.526979    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:05:40.869538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.869493    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" event={"ID":"3030a906-5209-413d-b120-c919c53ab069","Type":"ContainerStarted","Data":"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"}
Oct 30 22:05:40.869734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.869561    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" event={"ID":"3030a906-5209-413d-b120-c919c53ab069","Type":"ContainerStarted","Data":"4d655d0389af797af13d72c6b3b4b73fcab8913f1f4c27958237831627b96058"}
Oct 30 22:05:40.869734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.869631    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:40.992822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:40.992765    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" podStartSLOduration=1.992748891 podStartE2EDuration="1.992748891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:40.989848184 +0000 UTC m=+1664.163172610" watchObservedRunningTime="2026-10-30 22:05:40.992748891 +0000 UTC m=+1664.166073318"
Oct 30 22:05:48.580627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.580548    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"]
Oct 30 22:05:48.586473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.585956    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.640439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.640400    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"]
Oct 30 22:05:48.679890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.679847    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsvnk\" (UniqueName: \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.680100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.679982    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.680100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.680072    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.685990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.685945    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:05:48.686189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.686074    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:05:48.782012 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.781978    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.782217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.782042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.782275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.782254    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsvnk\" (UniqueName: \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.782814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.782786    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.784669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.784637    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.845023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.844920    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsvnk\" (UniqueName: \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\") pod \"maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:48.903180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:48.903129    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:49.313809 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:05:49.313768    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b456aed_e82e_413a_a120_3c0c6a5b4c2f.slice/crio-f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619 WatchSource:0}: Error finding container f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619: Status 404 returned error can't find the container with id f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619
Oct 30 22:05:49.317531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:49.317505    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"]
Oct 30 22:05:49.947551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:49.947451    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4" event={"ID":"1b456aed-e82e-413a-a120-3c0c6a5b4c2f","Type":"ContainerStarted","Data":"744e3d0be9d94d9153ebbebd654eb833fb1f99aaa36cb5efaba0e96ed68b09ea"}
Oct 30 22:05:49.947551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:49.947518    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4" event={"ID":"1b456aed-e82e-413a-a120-3c0c6a5b4c2f","Type":"ContainerStarted","Data":"f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619"}
Oct 30 22:05:50.067942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:50.067883    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4" podStartSLOduration=2.067863734 podStartE2EDuration="2.067863734s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:05:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:05:50.066643836 +0000 UTC m=+1673.239968265" watchObservedRunningTime="2026-10-30 22:05:50.067863734 +0000 UTC m=+1673.241188161"
Oct 30 22:05:50.815738 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:05:50.815687    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:05:50.957246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:50.957200    4882 generic.go:356] "Generic (PLEG): container finished" podID="1b456aed-e82e-413a-a120-3c0c6a5b4c2f" containerID="744e3d0be9d94d9153ebbebd654eb833fb1f99aaa36cb5efaba0e96ed68b09ea" exitCode=0
Oct 30 22:05:50.957659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:50.957287    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4" event={"ID":"1b456aed-e82e-413a-a120-3c0c6a5b4c2f","Type":"ContainerDied","Data":"744e3d0be9d94d9153ebbebd654eb833fb1f99aaa36cb5efaba0e96ed68b09ea"}
Oct 30 22:05:51.880542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:51.880514    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:05:52.152029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.151958    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:52.232992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.232955    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\") pod \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") "
Oct 30 22:05:52.233217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.233047    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\") pod \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") "
Oct 30 22:05:52.233217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.233149    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\" (UniqueName: \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\") pod \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\" (UID: \"1b456aed-e82e-413a-a120-3c0c6a5b4c2f\") "
Oct 30 22:05:52.233447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.233419    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca" pod "1b456aed-e82e-413a-a120-3c0c6a5b4c2f" (UID: "1b456aed-e82e-413a-a120-3c0c6a5b4c2f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:05:52.233793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.233774    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:52.235211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.235186    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca" pod "1b456aed-e82e-413a-a120-3c0c6a5b4c2f" (UID: "1b456aed-e82e-413a-a120-3c0c6a5b4c2f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:05:52.235321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.235301    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk" pod "1b456aed-e82e-413a-a120-3c0c6a5b4c2f" (UID: "1b456aed-e82e-413a-a120-3c0c6a5b4c2f"). InnerVolumeSpecName "kube-api-access-rsvnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:05:52.334928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.334889    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:52.334928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.334922    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsvnk\" (UniqueName: \"kubernetes.io/projected/1b456aed-e82e-413a-a120-3c0c6a5b4c2f-kube-api-access-rsvnk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:05:52.973708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.973674    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"
Oct 30 22:05:52.973903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.973704    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4" event={"ID":"1b456aed-e82e-413a-a120-3c0c6a5b4c2f","Type":"ContainerDied","Data":"f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619"}
Oct 30 22:05:52.973903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:52.973763    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f26cb6296d650daa4672c2f6246c254374eedf1bb6220b501145b7b409e77619"
Oct 30 22:05:57.400085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.400044    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 22:05:57.400529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.400273    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy" containerID="cri-o://20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f" gracePeriod=30
Oct 30 22:05:57.748199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.748104    4882 scope.go:122] "RemoveContainer" containerID="2d23271b266dbd11fbc30e4917f267af9c0336a6a1a04bc763c01ab37de7674b"
Oct 30 22:05:57.758772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.758751    4882 scope.go:122] "RemoveContainer" containerID="63b8fbdc847e60888754841394cb04e3e634d0172662c46fc0428f20bd816a3a"
Oct 30 22:05:57.768626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.768604    4882 scope.go:122] "RemoveContainer" containerID="5e5055192cfde1480be9fcb683715afdd8f9e03715d563c4ea6394e1bdc094b4"
Oct 30 22:05:57.778799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.778777    4882 scope.go:122] "RemoveContainer" containerID="e934ff984e9bb4f4453fbb088514eb72cc677c643ed869ba0a9cf306f4cfa1ed"
Oct 30 22:05:57.788399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.788375    4882 scope.go:122] "RemoveContainer" containerID="8584eed5c2e1b305d800beb49a6e29b36b3b409d8a198d51e4970dcac32598f2"
Oct 30 22:05:57.798525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.798498    4882 scope.go:122] "RemoveContainer" containerID="3c3f47ab11c8a0613ec0876a0ead75719597a79f54a177c4823911d40ba4b293"
Oct 30 22:05:57.808839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.808814    4882 scope.go:122] "RemoveContainer" containerID="a715f11a8adcba2e0d0abaca8c7d947911f7ba53371d3b84156b1d9f76bce690"
Oct 30 22:05:57.818492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:57.818471    4882 scope.go:122] "RemoveContainer" containerID="3e532c2350ccb9b2663458890d2f53de8260747a0dc96ae04891a455e7338e6c"
Oct 30 22:05:59.507878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:05:59.507828    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:06:02.650194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.650161    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:06:02.676825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.676751    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 22:06:02.783641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783599    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.783849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783655    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.783849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783693    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.783849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783756    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.783849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783822    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783955    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.783990    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784056    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784082    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784109    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:02.784131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784096    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\") pod \"0737f847-c26f-4fba-8790-cb5b7875e88f\" (UID: \"0737f847-c26f-4fba-8790-cb5b7875e88f\") "
Oct 30 22:06:02.784470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784315    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:02.784470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784410    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:02.784712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784683    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:02.784886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784862    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.784965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784892    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.784965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784908    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0737f847-c26f-4fba-8790-cb5b7875e88f-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.784965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784924    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.784965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.784938    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.786159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.786127    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:06:02.786305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.786281    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:06:02.786526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.786505    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "kube-api-access-gsv5k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:02.786571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.786555    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token" pod "0737f847-c26f-4fba-8790-cb5b7875e88f" (UID: "0737f847-c26f-4fba-8790-cb5b7875e88f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:02.886757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.886713    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gsv5k\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-kube-api-access-gsv5k\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.886757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.886750    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.886757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.886762    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:02.887002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:02.886771    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0737f847-c26f-4fba-8790-cb5b7875e88f-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:03.051457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.051421    4882 generic.go:356] "Generic (PLEG): container finished" podID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerID="20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f" exitCode=0
Oct 30 22:06:03.051683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.051476    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"
Oct 30 22:06:03.051683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.051486    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" event={"ID":"0737f847-c26f-4fba-8790-cb5b7875e88f","Type":"ContainerDied","Data":"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"}
Oct 30 22:06:03.051683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.051550    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2" event={"ID":"0737f847-c26f-4fba-8790-cb5b7875e88f","Type":"ContainerDied","Data":"005b8306cc1aa4a27340827dba6c689bf5a19c768a7d5b0564b4953d9dad495f"}
Oct 30 22:06:03.051683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.051576    4882 scope.go:122] "RemoveContainer" containerID="20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"
Oct 30 22:06:03.062532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.062505    4882 scope.go:122] "RemoveContainer" containerID="20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"
Oct 30 22:06:03.062902 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:06:03.062881    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f\": container with ID starting with 20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f not found: ID does not exist" containerID="20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"
Oct 30 22:06:03.062983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.062914    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f"} err="failed to get container status \"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f\": rpc error: code = NotFound desc = could not find container \"20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f\": container with ID starting with 20dbef2f768eb2b2362f72de8c8d4040949ddefcd23990d7603752c787cf950f not found: ID does not exist"
Oct 30 22:06:03.230865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.230772    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 22:06:03.239222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:03.239186    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-tsjp2"]
Oct 30 22:06:04.943535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:04.943491    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" path="/var/lib/kubelet/pods/0737f847-c26f-4fba-8790-cb5b7875e88f/volumes"
Oct 30 22:06:07.090196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.090157    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"]
Oct 30 22:06:07.090873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.090849    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1b456aed-e82e-413a-a120-3c0c6a5b4c2f" containerName="revoke-keys"
Oct 30 22:06:07.090924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.090874    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b456aed-e82e-413a-a120-3c0c6a5b4c2f" containerName="revoke-keys"
Oct 30 22:06:07.090924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.090900    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy"
Oct 30 22:06:07.090924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.090906    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy"
Oct 30 22:06:07.091033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.091024    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b456aed-e82e-413a-a120-3c0c6a5b4c2f" containerName="revoke-keys"
Oct 30 22:06:07.091068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.091039    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0737f847-c26f-4fba-8790-cb5b7875e88f" containerName="istio-proxy"
Oct 30 22:06:07.096183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.096157    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.139030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.138991    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.139176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.139058    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.139176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.139146    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cjcbj\" (UniqueName: \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.150309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.150271    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"]
Oct 30 22:06:07.195710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.195680    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:06:07.195896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.195849    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:06:07.240411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.240367    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.240656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.240433    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.240656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.240496    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cjcbj\" (UniqueName: \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.241153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.241133    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.242902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.242879    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.302693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.302651    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cjcbj\" (UniqueName: \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\") pod \"maas-api-revoke-keys-e2e-api-a-2d2f63-j629w\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.410493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.410402    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:07.651310 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.651267    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" podUID="9c19d747-4968-4dba-89da-4c2295833576" containerName="payload-pre-processing" containerID="cri-o://510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2" gracePeriod=25
Oct 30 22:06:07.791496 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:06:07.791460    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod285a05ec_0bad_451b_a4bc_7da9bb90be5f.slice/crio-8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2 WatchSource:0}: Error finding container 8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2: Status 404 returned error can't find the container with id 8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2
Oct 30 22:06:07.797611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.797560    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"]
Oct 30 22:06:07.928207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:07.928136    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:06:08.051295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.051254    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\") pod \"9c19d747-4968-4dba-89da-4c2295833576\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") "
Oct 30 22:06:08.051525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.051503    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\" (UniqueName: \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\") pod \"9c19d747-4968-4dba-89da-4c2295833576\" (UID: \"9c19d747-4968-4dba-89da-4c2295833576\") "
Oct 30 22:06:08.051732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.051709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume" pod "9c19d747-4968-4dba-89da-4c2295833576" (UID: "9c19d747-4968-4dba-89da-4c2295833576"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:08.052128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.052105    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c19d747-4968-4dba-89da-4c2295833576-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:08.054258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.054227    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4" pod "9c19d747-4968-4dba-89da-4c2295833576" (UID: "9c19d747-4968-4dba-89da-4c2295833576"). InnerVolumeSpecName "kube-api-access-flrn4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:08.089482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.089440    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w" event={"ID":"285a05ec-0bad-451b-a4bc-7da9bb90be5f","Type":"ContainerStarted","Data":"8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2"}
Oct 30 22:06:08.091222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.091184    4882 generic.go:356] "Generic (PLEG): container finished" podID="9c19d747-4968-4dba-89da-4c2295833576" containerID="510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2" exitCode=0
Oct 30 22:06:08.091699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.091224    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"
Oct 30 22:06:08.091699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.091265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" event={"ID":"9c19d747-4968-4dba-89da-4c2295833576","Type":"ContainerDied","Data":"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"}
Oct 30 22:06:08.091699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.091324    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl" event={"ID":"9c19d747-4968-4dba-89da-4c2295833576","Type":"ContainerDied","Data":"e9ee380476c0dc49172428a482eb1582e30466882707160836961a8ab1bf2849"}
Oct 30 22:06:08.091699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.091354    4882 scope.go:122] "RemoveContainer" containerID="510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"
Oct 30 22:06:08.102440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.102419    4882 scope.go:122] "RemoveContainer" containerID="510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"
Oct 30 22:06:08.102826 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:06:08.102805    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2\": container with ID starting with 510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2 not found: ID does not exist" containerID="510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"
Oct 30 22:06:08.102910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.102837    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2"} err="failed to get container status \"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2\": rpc error: code = NotFound desc = could not find container \"510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2\": container with ID starting with 510bc94cc48d216f2bed577be0d092431b9e03449f67fe985f2915605b02a2d2 not found: ID does not exist"
Oct 30 22:06:08.153567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.153530    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flrn4\" (UniqueName: \"kubernetes.io/projected/9c19d747-4968-4dba-89da-4c2295833576-kube-api-access-flrn4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:08.264758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.264717    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:06:08.277776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.269574    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-695f5c-57cf8874ff-kxqtl"]
Oct 30 22:06:08.944130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:08.944092    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c19d747-4968-4dba-89da-4c2295833576" path="/var/lib/kubelet/pods/9c19d747-4968-4dba-89da-4c2295833576/volumes"
Oct 30 22:06:09.099930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:09.099893    4882 generic.go:356] "Generic (PLEG): container finished" podID="285a05ec-0bad-451b-a4bc-7da9bb90be5f" containerID="8b1faa2987d8ea1e347ef8d8b712a3f93460b6bb338837e5ee223975582be5cb" exitCode=0
Oct 30 22:06:09.100428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:09.099950    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w" event={"ID":"285a05ec-0bad-451b-a4bc-7da9bb90be5f","Type":"ContainerDied","Data":"8b1faa2987d8ea1e347ef8d8b712a3f93460b6bb338837e5ee223975582be5cb"}
Oct 30 22:06:10.261174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.261145    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:10.383429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.383011    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\" (UniqueName: \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\") pod \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") "
Oct 30 22:06:10.383429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.383128    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\") pod \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") "
Oct 30 22:06:10.383429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.383188    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\") pod \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\" (UID: \"285a05ec-0bad-451b-a4bc-7da9bb90be5f\") "
Oct 30 22:06:10.384041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.383999    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca" pod "285a05ec-0bad-451b-a4bc-7da9bb90be5f" (UID: "285a05ec-0bad-451b-a4bc-7da9bb90be5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:10.386444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.386192    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj" pod "285a05ec-0bad-451b-a4bc-7da9bb90be5f" (UID: "285a05ec-0bad-451b-a4bc-7da9bb90be5f"). InnerVolumeSpecName "kube-api-access-cjcbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:10.386574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.386513    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca" pod "285a05ec-0bad-451b-a4bc-7da9bb90be5f" (UID: "285a05ec-0bad-451b-a4bc-7da9bb90be5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:06:10.484832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.484798    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjcbj\" (UniqueName: \"kubernetes.io/projected/285a05ec-0bad-451b-a4bc-7da9bb90be5f-kube-api-access-cjcbj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.484832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.484830    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.484832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.484841    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/285a05ec-0bad-451b-a4bc-7da9bb90be5f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:10.605516 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.605482    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:06:10.606168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606151    4882 cpu_manager.go:414] "containerMap: removing container" podUID="285a05ec-0bad-451b-a4bc-7da9bb90be5f" containerName="revoke-keys"
Oct 30 22:06:10.606221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606169    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="285a05ec-0bad-451b-a4bc-7da9bb90be5f" containerName="revoke-keys"
Oct 30 22:06:10.606221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606193    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9c19d747-4968-4dba-89da-4c2295833576" containerName="payload-pre-processing"
Oct 30 22:06:10.606221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606199    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c19d747-4968-4dba-89da-4c2295833576" containerName="payload-pre-processing"
Oct 30 22:06:10.606323 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606305    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="285a05ec-0bad-451b-a4bc-7da9bb90be5f" containerName="revoke-keys"
Oct 30 22:06:10.606323 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.606316    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c19d747-4968-4dba-89da-4c2295833576" containerName="payload-pre-processing"
Oct 30 22:06:10.610072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.610050    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.663505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.663407    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2f6bc1\""
Oct 30 22:06:10.663505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.663415    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2f6bc1-dockercfg-m4jl5\""
Oct 30 22:06:10.666490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.666458    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:06:10.687438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.687396    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.687680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.687653    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq68j\" (UniqueName: \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.751987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.751950    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:06:10.757176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.757152    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:10.789188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.789147    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq68j\" (UniqueName: \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.789358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.789302    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.790092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.790069    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.818282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.818241    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:06:10.853462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.853425    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq68j\" (UniqueName: \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\") pod \"payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.890998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.890952    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:10.891216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.891179    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4q4d\" (UniqueName: \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:10.924182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.924086    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:10.993998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.993770    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4q4d\" (UniqueName: \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:10.995776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.995201    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:10.996190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:10.996162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:11.056389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.056350    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4q4d\" (UniqueName: \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\") pod \"payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") " pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:11.072173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.072137    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:11.118779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.118688    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w" event={"ID":"285a05ec-0bad-451b-a4bc-7da9bb90be5f","Type":"ContainerDied","Data":"8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2"}
Oct 30 22:06:11.118779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.118737    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"
Oct 30 22:06:11.118779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.118750    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8ac6a50f1658d57092dbc1f26c3eddeb6a934c9a8337e6b3f259dcb92543dee2"
Oct 30 22:06:11.307799 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:06:11.307763    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf58d1df3_b26f_4bb0_b2c4_ab67479b82cc.slice/crio-186ee33df49a96ce7732efa36b54e09f502426a5c61c1939243b29199a00e8fe WatchSource:0}: Error finding container 186ee33df49a96ce7732efa36b54e09f502426a5c61c1939243b29199a00e8fe: Status 404 returned error can't find the container with id 186ee33df49a96ce7732efa36b54e09f502426a5c61c1939243b29199a00e8fe
Oct 30 22:06:11.312483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.312457    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:06:11.506885 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:06:11.506852    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc492f48e_31ba_41cf_a2c8_bbfa9e426fda.slice/crio-3e971a4a29199e8ecb3113ec05f1f4187d0da6446226766414bad2c96407329d WatchSource:0}: Error finding container 3e971a4a29199e8ecb3113ec05f1f4187d0da6446226766414bad2c96407329d: Status 404 returned error can't find the container with id 3e971a4a29199e8ecb3113ec05f1f4187d0da6446226766414bad2c96407329d
Oct 30 22:06:11.509065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:11.509041    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:06:12.127515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.127468    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" event={"ID":"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc","Type":"ContainerStarted","Data":"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"}
Oct 30 22:06:12.127740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.127581    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" event={"ID":"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc","Type":"ContainerStarted","Data":"186ee33df49a96ce7732efa36b54e09f502426a5c61c1939243b29199a00e8fe"}
Oct 30 22:06:12.128118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.127924    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:12.130181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.130148    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" event={"ID":"c492f48e-31ba-41cf-a2c8-bbfa9e426fda","Type":"ContainerStarted","Data":"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"}
Oct 30 22:06:12.130319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.130208    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" event={"ID":"c492f48e-31ba-41cf-a2c8-bbfa9e426fda","Type":"ContainerStarted","Data":"3e971a4a29199e8ecb3113ec05f1f4187d0da6446226766414bad2c96407329d"}
Oct 30 22:06:12.130319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.130238    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:12.248429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.248373    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" podStartSLOduration=2.248357562 podStartE2EDuration="2.248357562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:12.244662672 +0000 UTC m=+1695.417987098" watchObservedRunningTime="2026-10-30 22:06:12.248357562 +0000 UTC m=+1695.421681989"
Oct 30 22:06:12.361940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:12.361886    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" podStartSLOduration=2.361871759 podStartE2EDuration="2.361871759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:12.360949215 +0000 UTC m=+1695.534273642" watchObservedRunningTime="2026-10-30 22:06:12.361871759 +0000 UTC m=+1695.535196186"
Oct 30 22:06:18.666651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:18.666613    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:06:20.810266 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:06:20.810229    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:06:23.140766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:23.140734    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:06:23.141181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:23.140802    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:06:23.667318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:23.667253    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" podUID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" containerName="payload-processing" containerID="cri-o://406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7" gracePeriod=25
Oct 30 22:06:23.961789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:23.961760    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:06:24.066377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.066342    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\" (UniqueName: \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\") pod \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") "
Oct 30 22:06:24.066632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.066414    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\") pod \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\" (UID: \"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5\") "
Oct 30 22:06:24.066845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.066820    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume" pod "6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" (UID: "6ee9bc10-4b4f-45f9-b693-f5519f5f39d5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:06:24.067033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.067011    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:24.068681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.068661    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88" pod "6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" (UID: "6ee9bc10-4b4f-45f9-b693-f5519f5f39d5"). InnerVolumeSpecName "kube-api-access-pgv88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:06:24.168399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.168350    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pgv88\" (UniqueName: \"kubernetes.io/projected/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5-kube-api-access-pgv88\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:06:24.224577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.224491    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" containerID="406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7" exitCode=0
Oct 30 22:06:24.224577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.224546    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"
Oct 30 22:06:24.224577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.224564    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" event={"ID":"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5","Type":"ContainerDied","Data":"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"}
Oct 30 22:06:24.224831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.224660    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp" event={"ID":"6ee9bc10-4b4f-45f9-b693-f5519f5f39d5","Type":"ContainerDied","Data":"430778caf4b2b3b476d545d4673940c5253d8bd39ea45dbe0118dcdc3b21ef4f"}
Oct 30 22:06:24.224831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.224684    4882 scope.go:122] "RemoveContainer" containerID="406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"
Oct 30 22:06:24.236560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.236539    4882 scope.go:122] "RemoveContainer" containerID="406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"
Oct 30 22:06:24.236941 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:06:24.236919    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7\": container with ID starting with 406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7 not found: ID does not exist" containerID="406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"
Oct 30 22:06:24.236990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.236950    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7"} err="failed to get container status \"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7\": rpc error: code = NotFound desc = could not find container \"406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7\": container with ID starting with 406bfb932178ea2f620d3f94048da9a56fb697e007e940fa9277fb42e3a653e7 not found: ID does not exist"
Oct 30 22:06:24.394728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.394690    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:06:24.400302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.400269    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-2d2f63-7bcbc66bdf-z9gbp"]
Oct 30 22:06:24.949096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:24.949060    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" path="/var/lib/kubelet/pods/6ee9bc10-4b4f-45f9-b693-f5519f5f39d5/volumes"
Oct 30 22:06:50.810543 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:06:50.810503    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:06:51.868328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.868290    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:06:51.869001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.868985    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" containerName="payload-processing"
Oct 30 22:06:51.869052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.869002    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" containerName="payload-processing"
Oct 30 22:06:51.869125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.869116    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ee9bc10-4b4f-45f9-b693-f5519f5f39d5" containerName="payload-processing"
Oct 30 22:06:51.873572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.873553    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.927077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.927043    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-4c260505-openshift-default-dockercfg-k4h5b\""
Oct 30 22:06:51.930299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930269    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930318    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930406    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930458    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930580    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930667    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930762    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h28rn\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.930824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.930808    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:51.981784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:51.981748    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:06:52.031933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.031901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.031933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.031943    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032038    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032082    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h28rn\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032110    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032205    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032254    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032303    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032352    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032387    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032454    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032697    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.032852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.032821    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.033156 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.033136    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.034457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.034435    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.034869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.034849    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.094911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.094872    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.095065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.094996    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h28rn\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-vmqvj\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.189525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.189423    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:52.619254 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:06:52.619216    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5389c44c_105b_47ad_a1d1_b4e4aeb77506.slice/crio-e4616f2718eb201d6bcb1077fc75da80e5ff8061ded67187f78de7d150a846e1 WatchSource:0}: Error finding container e4616f2718eb201d6bcb1077fc75da80e5ff8061ded67187f78de7d150a846e1: Status 404 returned error can't find the container with id e4616f2718eb201d6bcb1077fc75da80e5ff8061ded67187f78de7d150a846e1
Oct 30 22:06:52.622238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.622202    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:06:52.622350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.622270    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:06:52.622350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.622301    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:06:52.623181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:52.623158    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:06:53.052274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:53.052167    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"]
Oct 30 22:06:53.055898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:53.055871    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-829c419d-5bvs6"]
Oct 30 22:06:53.458317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:53.458285    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" event={"ID":"5389c44c-105b-47ad-a1d1-b4e4aeb77506","Type":"ContainerStarted","Data":"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"}
Oct 30 22:06:53.458517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:53.458336    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" event={"ID":"5389c44c-105b-47ad-a1d1-b4e4aeb77506","Type":"ContainerStarted","Data":"e4616f2718eb201d6bcb1077fc75da80e5ff8061ded67187f78de7d150a846e1"}
Oct 30 22:06:53.584048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:53.583992    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podStartSLOduration=2.5839756400000002 podStartE2EDuration="2.58397564s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:06:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:06:53.579253449 +0000 UTC m=+1736.752577875" watchObservedRunningTime="2026-10-30 22:06:53.58397564 +0000 UTC m=+1736.757300066"
Oct 30 22:06:54.189677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:54.189633    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:54.191571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:54.191539    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:06:54.191753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:54.191612    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:06:54.959865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:54.959831    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6a73c1eb-e967-4c64-8b8b-5530e7fb190a" path="/var/lib/kubelet/pods/6a73c1eb-e967-4c64-8b8b-5530e7fb190a/volumes"
Oct 30 22:06:55.190823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:55.190782    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:06:55.191245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:55.190859    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:06:56.190668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:56.190627    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:06:56.190880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:56.190699    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:06:57.190265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:57.190222    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:06:57.190487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:57.190282    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:06:57.902895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:57.902857    4882 scope.go:122] "RemoveContainer" containerID="5b0c4d2b7ae2c36370c5c231d0f51ae2ea7821174ce75e4c929ac2aa66f7a7f6"
Oct 30 22:06:58.190858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:58.190774    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused" start-of-body=
Oct 30 22:06:58.190858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:58.190839    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.128:15021/healthz/ready\": dial tcp 10.132.0.128:15021: connect: connection refused"
Oct 30 22:06:59.214508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:59.214474    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:59.215467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:59.215444    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:06:59.216295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:06:59.216268    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:07:01.174693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:01.174655    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:07:01.175188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:01.174927    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy" containerID="cri-o://a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b" gracePeriod=30
Oct 30 22:07:02.050510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:02.050470    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"]
Oct 30 22:07:02.056510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:02.056480    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-172c9155-cpx7m"]
Oct 30 22:07:02.781953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:02.781913    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:07:02.944234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:02.944201    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43b43c12-d6dc-4462-8590-4b019c759962" path="/var/lib/kubelet/pods/43b43c12-d6dc-4462-8590-4b019c759962/volumes"
Oct 30 22:07:03.053201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:03.053117    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"]
Oct 30 22:07:03.103538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:03.103499    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-574de1fd-vslps"]
Oct 30 22:07:04.944609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:04.944560    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fcc13a4-836c-4024-b2f6-97bc1862b1e1" path="/var/lib/kubelet/pods/5fcc13a4-836c-4024-b2f6-97bc1862b1e1/volumes"
Oct 30 22:07:06.464768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.464699    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:07:06.534278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534242    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534281    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534347    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534420    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534458    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534485    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534528    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534620    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534657    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\") pod \"82fec621-76db-4ce6-b6cf-47522601622a\" (UID: \"82fec621-76db-4ce6-b6cf-47522601622a\") "
Oct 30 22:07:06.534791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.534670    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:06.535157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.535128    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:06.535411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.535256    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:06.535527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.535278    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:06.535641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.535309    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:06.535762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.535399    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.537056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.537028    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "kube-api-access-sc4s7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:06.537179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.537069    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:06.537246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.537184    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:07:06.537493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.537465    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo" pod "82fec621-76db-4ce6-b6cf-47522601622a" (UID: "82fec621-76db-4ce6-b6cf-47522601622a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:07:06.561179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.561147    4882 generic.go:356] "Generic (PLEG): container finished" podID="82fec621-76db-4ce6-b6cf-47522601622a" containerID="a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b" exitCode=0
Oct 30 22:07:06.561422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.561404    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" event={"ID":"82fec621-76db-4ce6-b6cf-47522601622a","Type":"ContainerDied","Data":"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"}
Oct 30 22:07:06.561519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.561494    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj" event={"ID":"82fec621-76db-4ce6-b6cf-47522601622a","Type":"ContainerDied","Data":"cc817a718ab1d021761baa171d4651bdd6077742c6718b47b443879297147568"}
Oct 30 22:07:06.561575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.561531    4882 scope.go:122] "RemoveContainer" containerID="a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"
Oct 30 22:07:06.561749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.561733    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"
Oct 30 22:07:06.591565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.591524    4882 scope.go:122] "RemoveContainer" containerID="a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"
Oct 30 22:07:06.591962 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:06.591942    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b\": container with ID starting with a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b not found: ID does not exist" containerID="a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"
Oct 30 22:07:06.592030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.591971    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b"} err="failed to get container status \"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b\": rpc error: code = NotFound desc = could not find container \"a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b\": container with ID starting with a5b192b410055bbe4188fc8038afaaa5261ca7e6055526a8e36943753af9361b not found: ID does not exist"
Oct 30 22:07:06.637407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637372    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637404    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82fec621-76db-4ce6-b6cf-47522601622a-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637421    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82fec621-76db-4ce6-b6cf-47522601622a-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637435    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637450    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637463    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637475    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82fec621-76db-4ce6-b6cf-47522601622a-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.637688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.637489    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc4s7\" (UniqueName: \"kubernetes.io/projected/82fec621-76db-4ce6-b6cf-47522601622a-kube-api-access-sc4s7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:06.758331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.758243    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:07:06.761872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.761841    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-9v9mj"]
Oct 30 22:07:06.944809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:06.944771    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82fec621-76db-4ce6-b6cf-47522601622a" path="/var/lib/kubelet/pods/82fec621-76db-4ce6-b6cf-47522601622a/volumes"
Oct 30 22:07:07.013958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.013872    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:07:07.014568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.014552    4882 cpu_manager.go:414] "containerMap: removing container" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy"
Oct 30 22:07:07.014667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.014570    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy"
Oct 30 22:07:07.014709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.014699    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="82fec621-76db-4ce6-b6cf-47522601622a" containerName="istio-proxy"
Oct 30 22:07:07.019991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.019970    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.090994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.090961    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:07:07.144874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.144839    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.144917    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.144967    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145134    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145192    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145228    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lxq8b\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145264    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145346    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.145398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.145386    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247117    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247177    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247199    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lxq8b\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247283    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247345    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247389    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247455    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247619    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247769    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247798    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.247937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.247867    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.248186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.248167    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.249720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.249694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.250026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.250010    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.309014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.308935    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.309182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.309155    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lxq8b\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-c8mgv\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.335998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.335961    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:07.767008 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:07:07.766977    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660b77f6_9528_467e_9019_a272dcbca1b0.slice/crio-7aa08dcf3b177feaee09106324f6a573b921e1115f9b1029868ba12980d3fdf8 WatchSource:0}: Error finding container 7aa08dcf3b177feaee09106324f6a573b921e1115f9b1029868ba12980d3fdf8: Status 404 returned error can't find the container with id 7aa08dcf3b177feaee09106324f6a573b921e1115f9b1029868ba12980d3fdf8
Oct 30 22:07:07.769114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.769090    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:07:07.769816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.769788    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:07.769876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.769857    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:07.769930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:07.769893    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:08.578798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:08.578759    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" event={"ID":"660b77f6-9528-467e-9019-a272dcbca1b0","Type":"ContainerStarted","Data":"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"}
Oct 30 22:07:08.578983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:08.578813    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" event={"ID":"660b77f6-9528-467e-9019-a272dcbca1b0","Type":"ContainerStarted","Data":"7aa08dcf3b177feaee09106324f6a573b921e1115f9b1029868ba12980d3fdf8"}
Oct 30 22:07:08.839675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:08.824872    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podStartSLOduration=2.824853091 podStartE2EDuration="2.824853091s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:08.770109118 +0000 UTC m=+1751.943433547" watchObservedRunningTime="2026-10-30 22:07:08.824853091 +0000 UTC m=+1751.998177526"
Oct 30 22:07:09.347673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:09.338005    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:09.347673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:09.347240    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:07:09.347673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:09.347302    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:07:10.343614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:10.337068    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:07:10.343614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:10.337143    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:07:11.349616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:11.348950    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:07:11.349616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:11.349015    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:07:12.349735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:12.349694    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:07:12.352132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:12.349765    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:07:13.350143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:13.350105    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused" start-of-body=
Oct 30 22:07:13.350848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:13.350812    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.129:15021/healthz/ready\": dial tcp 10.132.0.129:15021: connect: connection refused"
Oct 30 22:07:14.341748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:14.341707    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:14.342131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:14.342103    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:14.342845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:14.342826    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:07:20.809402 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:20.809361    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:07:22.116820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.116783    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:07:22.121030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.121004    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.177907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.177870    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178061    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178139    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178276    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178329    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178381    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178427    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178474    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.178542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.178530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2r9qk\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.216447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.216406    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:07:22.280076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280023    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280087    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280113    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280170    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280191    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280259    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2r9qk\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280322    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280478    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280508    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280534    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.280870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280844    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.281009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.280984    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.281094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.281077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.281135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.281092    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.282609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.282566    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.282728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.282709    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.345244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.345205    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2r9qk\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.345711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.345687    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\") pod \"e2e-mt-4c260505-openshift-default-6f77466685-lg8h5\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") " pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.436130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.436033    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:22.867879 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:07:22.867846    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod011c4c48_7879_4f20_9c99_ba72521e2096.slice/crio-83953eddb9406a0a4312c730eecf95f47ebdcf54f7b168c344627725aee97c58 WatchSource:0}: Error finding container 83953eddb9406a0a4312c730eecf95f47ebdcf54f7b168c344627725aee97c58: Status 404 returned error can't find the container with id 83953eddb9406a0a4312c730eecf95f47ebdcf54f7b168c344627725aee97c58
Oct 30 22:07:22.870102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.870072    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:07:22.870813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.870776    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:22.870916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.870850    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:22.870916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:22.870884    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:07:23.684371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:23.684331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" event={"ID":"011c4c48-7879-4f20-9c99-ba72521e2096","Type":"ContainerStarted","Data":"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"}
Oct 30 22:07:23.684801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:23.684397    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" event={"ID":"011c4c48-7879-4f20-9c99-ba72521e2096","Type":"ContainerStarted","Data":"83953eddb9406a0a4312c730eecf95f47ebdcf54f7b168c344627725aee97c58"}
Oct 30 22:07:23.862006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:23.861950    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podStartSLOduration=1.861934657 podStartE2EDuration="1.861934657s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:23.859245963 +0000 UTC m=+1767.032570390" watchObservedRunningTime="2026-10-30 22:07:23.861934657 +0000 UTC m=+1767.035259142"
Oct 30 22:07:24.442620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:24.437983    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:24.449636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:24.443231    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:24.449636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:24.443300    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:25.437614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:25.436976    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:25.437614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:25.437050    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:26.092182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.092137    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-ad1e97"]
Oct 30 22:07:26.096658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.096631    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:26.166164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.163674    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-ad1e97"]
Oct 30 22:07:26.175860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.175826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2znc\" (UniqueName: \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\") pod \"test-curl-7652-ad1e97\" (UID: \"ed75514e-c43a-4c15-b854-f67fda4b356b\") " pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:26.193497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.193449    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jppg\""
Oct 30 22:07:26.277448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.277411    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2znc\" (UniqueName: \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\") pod \"test-curl-7652-ad1e97\" (UID: \"ed75514e-c43a-4c15-b854-f67fda4b356b\") " pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:26.357665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.357575    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2znc\" (UniqueName: \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\") pod \"test-curl-7652-ad1e97\" (UID: \"ed75514e-c43a-4c15-b854-f67fda4b356b\") " pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:26.412343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.412305    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:26.449610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.446333    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:26.449610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.446397    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:26.823656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:26.823622    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-ad1e97"]
Oct 30 22:07:27.060168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.060131    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"]
Oct 30 22:07:27.064941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.064910    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5b80383f-6qvb7"]
Oct 30 22:07:27.441220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.440029    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:27.441220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.440101    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:27.718490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.718391    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-ad1e97" event={"ID":"ed75514e-c43a-4c15-b854-f67fda4b356b","Type":"ContainerStarted","Data":"5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2"}
Oct 30 22:07:27.718490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.718458    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-ad1e97" event={"ID":"ed75514e-c43a-4c15-b854-f67fda4b356b","Type":"ContainerStarted","Data":"aeff83b13eb3d2cd30efd855176e1190d7e7004dcadc4abc8c6b55bcb2343a20"}
Oct 30 22:07:27.854241 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.854163    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7652-ad1e97" podStartSLOduration=1.8541460829999998 podStartE2EDuration="1.854146083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:27.8488182 +0000 UTC m=+1771.022142628" watchObservedRunningTime="2026-10-30 22:07:27.854146083 +0000 UTC m=+1771.027470504"
Oct 30 22:07:27.952829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.952785    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"]
Oct 30 22:07:27.956453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:27.956416    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-695f5c-bz7q4"]
Oct 30 22:07:28.457686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:28.440990    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:28.457686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:28.441109    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:28.956550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:28.956510    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b456aed-e82e-413a-a120-3c0c6a5b4c2f" path="/var/lib/kubelet/pods/1b456aed-e82e-413a-a120-3c0c6a5b4c2f/volumes"
Oct 30 22:07:28.957071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:28.957054    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60658c6f-1783-4494-9685-0c73240f6ffc" path="/var/lib/kubelet/pods/60658c6f-1783-4494-9685-0c73240f6ffc/volumes"
Oct 30 22:07:29.437013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.436976    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused" start-of-body=
Oct 30 22:07:29.437256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.437051    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.130:15021/healthz/ready\": dial tcp 10.132.0.130:15021: connect: connection refused"
Oct 30 22:07:29.441643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.439996    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-ad1e97"]
Oct 30 22:07:29.445444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.445408    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-ad1e97"]
Oct 30 22:07:29.735409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.735276    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7652-ad1e97" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" containerName="test-curl-7652-ad1e97" containerID="cri-o://5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2" gracePeriod=2
Oct 30 22:07:29.787108 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:29.787061    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-ad1e97\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:29.977663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.977619    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"]
Oct 30 22:07:29.978519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.978485    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" containerName="test-curl-7652-ad1e97"
Oct 30 22:07:29.978519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.978515    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" containerName="test-curl-7652-ad1e97"
Oct 30 22:07:29.978799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.978782    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" containerName="test-curl-7652-ad1e97"
Oct 30 22:07:29.983760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:29.983733    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.045152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.044634    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.045152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.044780    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6rf4\" (UniqueName: \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.045152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.044853    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.049621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.049531    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"]
Oct 30 22:07:30.083820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.083780    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:30.083820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.083782    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:07:30.146293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.146261    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.146507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.146349    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6rf4\" (UniqueName: \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.146507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.146410    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.146968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.146946    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.148822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.148797    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.165018 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:30.164980    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-ad1e97\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:30.208674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.208643    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6rf4\" (UniqueName: \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\") pod \"maas-api-revoke-keys-e2e-mt-4c260505-k4nvt\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.298930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.298847    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:30.441222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.441184    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:30.441555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.441535    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:30.442168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.442148    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:07:30.492971 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:30.492933    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-ad1e97\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:30.676458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.676425    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"]
Oct 30 22:07:30.708688 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:07:30.708652    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b4d8689_eb18_4c55_b13a_2d7a4199e309.slice/crio-c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c WatchSource:0}: Error finding container c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c: Status 404 returned error can't find the container with id c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c
Oct 30 22:07:30.760898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:30.760747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt" event={"ID":"3b4d8689-eb18-4c55-b13a-2d7a4199e309","Type":"ContainerStarted","Data":"c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c"}
Oct 30 22:07:30.787166 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:30.787127    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-ad1e97\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:31.769928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:31.769800    4882 generic.go:356] "Generic (PLEG): container finished" podID="3b4d8689-eb18-4c55-b13a-2d7a4199e309" containerID="c8b9dadb5e6dfd15344788479a4b9360f873bcf8e02d187721c36efb6cd8d67b" exitCode=0
Oct 30 22:07:31.769928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:31.769890    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt" event={"ID":"3b4d8689-eb18-4c55-b13a-2d7a4199e309","Type":"ContainerDied","Data":"c8b9dadb5e6dfd15344788479a4b9360f873bcf8e02d187721c36efb6cd8d67b"}
Oct 30 22:07:31.984227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:31.984161    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:32.075178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.075132    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\" (UniqueName: \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\") pod \"ed75514e-c43a-4c15-b854-f67fda4b356b\" (UID: \"ed75514e-c43a-4c15-b854-f67fda4b356b\") "
Oct 30 22:07:32.077351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.077323    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc" pod "ed75514e-c43a-4c15-b854-f67fda4b356b" (UID: "ed75514e-c43a-4c15-b854-f67fda4b356b"). InnerVolumeSpecName "kube-api-access-n2znc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:32.176994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.176955    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2znc\" (UniqueName: \"kubernetes.io/projected/ed75514e-c43a-4c15-b854-f67fda4b356b-kube-api-access-n2znc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:32.778504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.778461    4882 generic.go:356] "Generic (PLEG): container finished" podID="ed75514e-c43a-4c15-b854-f67fda4b356b" containerID="5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2" exitCode=137
Oct 30 22:07:32.779043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.778521    4882 scope.go:122] "RemoveContainer" containerID="5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2"
Oct 30 22:07:32.779043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.778526    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:32.794967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.794945    4882 scope.go:122] "RemoveContainer" containerID="5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2"
Oct 30 22:07:32.795686 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:32.795558    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2\": container with ID starting with 5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2 not found: ID does not exist" containerID="5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2"
Oct 30 22:07:32.795686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.795628    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2"} err="failed to get container status \"5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2\": rpc error: code = NotFound desc = could not find container \"5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2\": container with ID starting with 5246b85200d121bbc67bc754392c01c667eeaaeb26b12f31823504dd1711a6f2 not found: ID does not exist"
Oct 30 22:07:32.841554 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:32.841507    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-ad1e97\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" pod="opendatahub/test-curl-7652-ad1e97"
Oct 30 22:07:32.946139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.946107    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed75514e-c43a-4c15-b854-f67fda4b356b" path="/var/lib/kubelet/pods/ed75514e-c43a-4c15-b854-f67fda4b356b/volumes"
Oct 30 22:07:32.978545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:32.978515    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:33.088332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.088292    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\") pod \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") "
Oct 30 22:07:33.088332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.088338    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\" (UniqueName: \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\") pod \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") "
Oct 30 22:07:33.088765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.088476    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\") pod \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\" (UID: \"3b4d8689-eb18-4c55-b13a-2d7a4199e309\") "
Oct 30 22:07:33.088986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.088958    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca" pod "3b4d8689-eb18-4c55-b13a-2d7a4199e309" (UID: "3b4d8689-eb18-4c55-b13a-2d7a4199e309"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:33.089959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.089921    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:33.091390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.091366    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca" pod "3b4d8689-eb18-4c55-b13a-2d7a4199e309" (UID: "3b4d8689-eb18-4c55-b13a-2d7a4199e309"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:33.091601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.091562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4" pod "3b4d8689-eb18-4c55-b13a-2d7a4199e309" (UID: "3b4d8689-eb18-4c55-b13a-2d7a4199e309"). InnerVolumeSpecName "kube-api-access-s6rf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:33.191292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.191255    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s6rf4\" (UniqueName: \"kubernetes.io/projected/3b4d8689-eb18-4c55-b13a-2d7a4199e309-kube-api-access-s6rf4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:33.191292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.191286    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3b4d8689-eb18-4c55-b13a-2d7a4199e309-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:33.787524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.787491    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"
Oct 30 22:07:33.787937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.787524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt" event={"ID":"3b4d8689-eb18-4c55-b13a-2d7a4199e309","Type":"ContainerDied","Data":"c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c"}
Oct 30 22:07:33.787937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:33.787574    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c7dd52d1b26a5ff03ced3311d7fe2f440ce7b4d0af68b339170ce8c43155e75c"
Oct 30 22:07:35.809863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.809816    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"]
Oct 30 22:07:35.810810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.810782    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3b4d8689-eb18-4c55-b13a-2d7a4199e309" containerName="revoke-keys"
Oct 30 22:07:35.811005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.810979    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3b4d8689-eb18-4c55-b13a-2d7a4199e309" containerName="revoke-keys"
Oct 30 22:07:35.811273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.811255    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3b4d8689-eb18-4c55-b13a-2d7a4199e309" containerName="revoke-keys"
Oct 30 22:07:35.815990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.815965    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:35.868657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.868626    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"]
Oct 30 22:07:35.914215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.914180    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:35.914419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.914269    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:07:35.931962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.931927    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:35.932151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.932129    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:35.932338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:35.932309    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrvcq\" (UniqueName: \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.033876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.033840    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.034042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.033995    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrvcq\" (UniqueName: \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.034042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.034025    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.034518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.034487    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.036460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.036441    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.096092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.096057    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrvcq\" (UniqueName: \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\") pod \"maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.131475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.131436    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:36.537220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.537190    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"]
Oct 30 22:07:36.812721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:36.812635    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf" event={"ID":"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f","Type":"ContainerStarted","Data":"6f6319a6b5b9430f7fd5fb499e3cb09ccd7f77e2462fa46d5068a640e5ead49f"}
Oct 30 22:07:37.822425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:37.822334    4882 generic.go:356] "Generic (PLEG): container finished" podID="fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" containerID="7e18972bdfef49747c7736676693a37acaf07d6de4475c752cd07da339c039e5" exitCode=0
Oct 30 22:07:37.822858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:37.822420    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf" event={"ID":"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f","Type":"ContainerDied","Data":"7e18972bdfef49747c7736676693a37acaf07d6de4475c752cd07da339c039e5"}
Oct 30 22:07:39.005076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.005044    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:39.178306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.178219    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\" (UniqueName: \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\") pod \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") "
Oct 30 22:07:39.178455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.178426    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\") pod \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") "
Oct 30 22:07:39.178503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.178454    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\") pod \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\" (UID: \"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f\") "
Oct 30 22:07:39.178891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.178865    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca" pod "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" (UID: "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:39.180475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.180451    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq" pod "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" (UID: "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f"). InnerVolumeSpecName "kube-api-access-xrvcq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:39.180542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.180460    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca" pod "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" (UID: "fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:39.280851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.280820    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrvcq\" (UniqueName: \"kubernetes.io/projected/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-kube-api-access-xrvcq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:39.280851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.280850    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:39.281065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.280860    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:39.839964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.839935    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"
Oct 30 22:07:39.840176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.839960    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf" event={"ID":"fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f","Type":"ContainerDied","Data":"6f6319a6b5b9430f7fd5fb499e3cb09ccd7f77e2462fa46d5068a640e5ead49f"}
Oct 30 22:07:39.840176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:39.840018    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6f6319a6b5b9430f7fd5fb499e3cb09ccd7f77e2462fa46d5068a640e5ead49f"
Oct 30 22:07:40.811072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.811032    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"]
Oct 30 22:07:40.811715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.811697    4882 cpu_manager.go:414] "containerMap: removing container" podUID="fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" containerName="revoke-keys"
Oct 30 22:07:40.811759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.811716    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" containerName="revoke-keys"
Oct 30 22:07:40.811955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.811939    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" containerName="revoke-keys"
Oct 30 22:07:40.815470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.815453    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:40.873706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.873666    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"]
Oct 30 22:07:40.901359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.901309    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:40.901554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.901497    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4mz8p\" (UniqueName: \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:40.901723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.901697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:40.919510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.919478    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:40.919743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:40.919516    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:07:41.002617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.002545    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4mz8p\" (UniqueName: \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.002962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.002928    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.003132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.003111    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.003554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.003530    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.005390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.005374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.065249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.065169    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4mz8p\" (UniqueName: \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\") pod \"maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.133758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.133718    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:41.517134 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:07:41.517098    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19ac61fa_88e5_4c36_b26f_73fa556f5516.slice/crio-fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16 WatchSource:0}: Error finding container fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16: Status 404 returned error can't find the container with id fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16
Oct 30 22:07:41.518312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.518285    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"]
Oct 30 22:07:41.856732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.856702    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr" event={"ID":"19ac61fa-88e5-4c36-b26f-73fa556f5516","Type":"ContainerStarted","Data":"36fbb4f71d4e770428fbfcaaf42b79748c215453de58a7f1eb5c8fa537b45c6d"}
Oct 30 22:07:41.857096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.856753    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr" event={"ID":"19ac61fa-88e5-4c36-b26f-73fa556f5516","Type":"ContainerStarted","Data":"fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16"}
Oct 30 22:07:41.975505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:41.975447    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr" podStartSLOduration=1.975432834 podStartE2EDuration="1.975432834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:07:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:07:41.972706169 +0000 UTC m=+1785.146030596" watchObservedRunningTime="2026-10-30 22:07:41.975432834 +0000 UTC m=+1785.148757260"
Oct 30 22:07:42.866460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:42.866424    4882 generic.go:356] "Generic (PLEG): container finished" podID="19ac61fa-88e5-4c36-b26f-73fa556f5516" containerID="36fbb4f71d4e770428fbfcaaf42b79748c215453de58a7f1eb5c8fa537b45c6d" exitCode=0
Oct 30 22:07:42.866896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:42.866519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr" event={"ID":"19ac61fa-88e5-4c36-b26f-73fa556f5516","Type":"ContainerDied","Data":"36fbb4f71d4e770428fbfcaaf42b79748c215453de58a7f1eb5c8fa537b45c6d"}
Oct 30 22:07:42.896858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:42.896807    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:07:44.043190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.043163    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:44.150072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.149988    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\" (UniqueName: \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\") pod \"19ac61fa-88e5-4c36-b26f-73fa556f5516\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") "
Oct 30 22:07:44.150231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.150091    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\") pod \"19ac61fa-88e5-4c36-b26f-73fa556f5516\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") "
Oct 30 22:07:44.150231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.150186    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\") pod \"19ac61fa-88e5-4c36-b26f-73fa556f5516\" (UID: \"19ac61fa-88e5-4c36-b26f-73fa556f5516\") "
Oct 30 22:07:44.150623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.150562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca" pod "19ac61fa-88e5-4c36-b26f-73fa556f5516" (UID: "19ac61fa-88e5-4c36-b26f-73fa556f5516"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:44.150849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.150829    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.152230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.152203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p" pod "19ac61fa-88e5-4c36-b26f-73fa556f5516" (UID: "19ac61fa-88e5-4c36-b26f-73fa556f5516"). InnerVolumeSpecName "kube-api-access-4mz8p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:44.152321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.152210    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca" pod "19ac61fa-88e5-4c36-b26f-73fa556f5516" (UID: "19ac61fa-88e5-4c36-b26f-73fa556f5516"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:07:44.252167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.252118    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4mz8p\" (UniqueName: \"kubernetes.io/projected/19ac61fa-88e5-4c36-b26f-73fa556f5516-kube-api-access-4mz8p\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.252167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.252157    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/19ac61fa-88e5-4c36-b26f-73fa556f5516-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:44.881517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.881487    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"
Oct 30 22:07:44.881752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.881490    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr" event={"ID":"19ac61fa-88e5-4c36-b26f-73fa556f5516","Type":"ContainerDied","Data":"fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16"}
Oct 30 22:07:44.881752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:44.881646    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa9b4457de75d5b730e149ea9e3f443f3cd2a051f35c4a5f9fb77d765f710d16"
Oct 30 22:07:47.320301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:47.320265    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:07:47.428815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:47.428768    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:07:47.897223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:47.897162    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" podUID="3030a906-5209-413d-b120-c919c53ab069" containerName="payload-pre-processing" containerID="cri-o://2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c" gracePeriod=25
Oct 30 22:07:48.174387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.174357    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:07:48.320298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.320251    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\" (UniqueName: \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\") pod \"3030a906-5209-413d-b120-c919c53ab069\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") "
Oct 30 22:07:48.320880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.320312    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\") pod \"3030a906-5209-413d-b120-c919c53ab069\" (UID: \"3030a906-5209-413d-b120-c919c53ab069\") "
Oct 30 22:07:48.336770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.329967    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume" pod "3030a906-5209-413d-b120-c919c53ab069" (UID: "3030a906-5209-413d-b120-c919c53ab069"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:48.361097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.361051    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl" pod "3030a906-5209-413d-b120-c919c53ab069" (UID: "3030a906-5209-413d-b120-c919c53ab069"). InnerVolumeSpecName "kube-api-access-q44gl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:48.422253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.422172    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q44gl\" (UniqueName: \"kubernetes.io/projected/3030a906-5209-413d-b120-c919c53ab069-kube-api-access-q44gl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:48.422253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.422204    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3030a906-5209-413d-b120-c919c53ab069-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:48.935564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.935518    4882 generic.go:356] "Generic (PLEG): container finished" podID="3030a906-5209-413d-b120-c919c53ab069" containerID="2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c" exitCode=0
Oct 30 22:07:48.935753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.935669    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"
Oct 30 22:07:48.946794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.946522    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" event={"ID":"3030a906-5209-413d-b120-c919c53ab069","Type":"ContainerDied","Data":"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"}
Oct 30 22:07:48.946794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.946605    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j" event={"ID":"3030a906-5209-413d-b120-c919c53ab069","Type":"ContainerDied","Data":"4d655d0389af797af13d72c6b3b4b73fcab8913f1f4c27958237831627b96058"}
Oct 30 22:07:48.946794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.946639    4882 scope.go:122] "RemoveContainer" containerID="2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"
Oct 30 22:07:48.958876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.958853    4882 scope.go:122] "RemoveContainer" containerID="2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"
Oct 30 22:07:48.959204 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:48.959178    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c\": container with ID starting with 2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c not found: ID does not exist" containerID="2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"
Oct 30 22:07:48.959294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:48.959213    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c"} err="failed to get container status \"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c\": rpc error: code = NotFound desc = could not find container \"2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c\": container with ID starting with 2ca7fdc8aae8c0a224d6c83c5b5dd08905e3a3d3c51d986f52c28622a91f398c not found: ID does not exist"
Oct 30 22:07:49.132784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:49.132753    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:07:49.181511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:49.181478    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-4c260505-65bb6f8647-f6h7j"]
Oct 30 22:07:49.970113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:49.970073    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"]
Oct 30 22:07:50.020335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:50.020301    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-2d2f63-j629w"]
Oct 30 22:07:50.811318 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:50.811285    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:07:50.946783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:50.946739    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="285a05ec-0bad-451b-a4bc-7da9bb90be5f" path="/var/lib/kubelet/pods/285a05ec-0bad-451b-a4bc-7da9bb90be5f/volumes"
Oct 30 22:07:50.947451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:50.947418    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3030a906-5209-413d-b120-c919c53ab069" path="/var/lib/kubelet/pods/3030a906-5209-413d-b120-c919c53ab069/volumes"
Oct 30 22:07:52.318743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.318710    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:07:52.321494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.321462    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" podUID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" containerName="payload-processing" containerID="cri-o://f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019" gracePeriod=25
Oct 30 22:07:52.429415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.429334    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" podUID="9cdd56c6-601c-4b2a-812f-880713e7622e" containerName="payload-pre-processing" containerID="cri-o://505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2" gracePeriod=25
Oct 30 22:07:52.434726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.434614    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:07:52.604414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.604382    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:07:52.681679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.681639    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\" (UniqueName: \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\") pod \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") "
Oct 30 22:07:52.681943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.681921    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\") pod \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\" (UID: \"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452\") "
Oct 30 22:07:52.682538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.682501    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume" pod "8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" (UID: "8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:52.682769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.682729    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:52.686914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.686882    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn" pod "8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" (UID: "8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452"). InnerVolumeSpecName "kube-api-access-vdsxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:52.720720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.720687    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:07:52.789573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.789533    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vdsxn\" (UniqueName: \"kubernetes.io/projected/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452-kube-api-access-vdsxn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:52.890184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.890147    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\") pod \"9cdd56c6-601c-4b2a-812f-880713e7622e\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") "
Oct 30 22:07:52.890396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.890301    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\" (UniqueName: \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\") pod \"9cdd56c6-601c-4b2a-812f-880713e7622e\" (UID: \"9cdd56c6-601c-4b2a-812f-880713e7622e\") "
Oct 30 22:07:52.890524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.890498    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume" pod "9cdd56c6-601c-4b2a-812f-880713e7622e" (UID: "9cdd56c6-601c-4b2a-812f-880713e7622e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:52.890721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.890704    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9cdd56c6-601c-4b2a-812f-880713e7622e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:52.892549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.892482    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v" pod "9cdd56c6-601c-4b2a-812f-880713e7622e" (UID: "9cdd56c6-601c-4b2a-812f-880713e7622e"). InnerVolumeSpecName "kube-api-access-htd2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:52.976122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.976087    4882 generic.go:356] "Generic (PLEG): container finished" podID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" containerID="f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019" exitCode=0
Oct 30 22:07:52.976318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.976123    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" event={"ID":"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452","Type":"ContainerDied","Data":"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"}
Oct 30 22:07:52.976318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.976167    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"
Oct 30 22:07:52.976318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.976182    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6" event={"ID":"8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452","Type":"ContainerDied","Data":"778fae5d1345552ac560298b25853f3fcc470a53b274cd8bdb0a741fccffa28b"}
Oct 30 22:07:52.976318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.976205    4882 scope.go:122] "RemoveContainer" containerID="f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"
Oct 30 22:07:52.978266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.978240    4882 generic.go:356] "Generic (PLEG): container finished" podID="9cdd56c6-601c-4b2a-812f-880713e7622e" containerID="505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2" exitCode=0
Oct 30 22:07:52.978398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.978332    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" event={"ID":"9cdd56c6-601c-4b2a-812f-880713e7622e","Type":"ContainerDied","Data":"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"}
Oct 30 22:07:52.978398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.978383    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f" event={"ID":"9cdd56c6-601c-4b2a-812f-880713e7622e","Type":"ContainerDied","Data":"bc6d54eb677f8e1d675a9f4ffda96ac5c9ad6d7da3621549374a7746b6539b51"}
Oct 30 22:07:52.978511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.978481    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"
Oct 30 22:07:52.989392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.989367    4882 scope.go:122] "RemoveContainer" containerID="f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"
Oct 30 22:07:52.989840 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:52.989770    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019\": container with ID starting with f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019 not found: ID does not exist" containerID="f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"
Oct 30 22:07:52.989840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.989804    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019"} err="failed to get container status \"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019\": rpc error: code = NotFound desc = could not find container \"f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019\": container with ID starting with f05f116c7f6f6059f27f34d268664e60028695d81f7998daa73fcc027f276019 not found: ID does not exist"
Oct 30 22:07:52.989840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.989825    4882 scope.go:122] "RemoveContainer" containerID="505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"
Oct 30 22:07:52.991572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:52.991546    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-htd2v\" (UniqueName: \"kubernetes.io/projected/9cdd56c6-601c-4b2a-812f-880713e7622e-kube-api-access-htd2v\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:53.004192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.004167    4882 scope.go:122] "RemoveContainer" containerID="505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"
Oct 30 22:07:53.004678 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:53.004646    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2\": container with ID starting with 505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2 not found: ID does not exist" containerID="505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"
Oct 30 22:07:53.004791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.004696    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2"} err="failed to get container status \"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2\": rpc error: code = NotFound desc = could not find container \"505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2\": container with ID starting with 505438dbea5252e4b4fbc9bbe7d85d602f811e9352d9e6b42376705a219871f2 not found: ID does not exist"
Oct 30 22:07:53.144560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.144476    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:07:53.148732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.148702    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a5f7b9-5b5cfccf87-sq46f"]
Oct 30 22:07:53.257981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.257944    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:07:53.264742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:53.264710    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a5f7b9-6bfb557b65-2w4q6"]
Oct 30 22:07:54.945300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:54.945265    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" path="/var/lib/kubelet/pods/8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452/volumes"
Oct 30 22:07:54.945771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:54.945755    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9cdd56c6-601c-4b2a-812f-880713e7622e" path="/var/lib/kubelet/pods/9cdd56c6-601c-4b2a-812f-880713e7622e/volumes"
Oct 30 22:07:57.176033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.175996    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:07:57.185352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.185322    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:07:57.319110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.319036    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" podUID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" containerName="payload-processing" containerID="cri-o://83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f" gracePeriod=25
Oct 30 22:07:57.435633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.435549    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" podUID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" containerName="payload-pre-processing" containerID="cri-o://9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd" gracePeriod=25
Oct 30 22:07:57.605212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.605180    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:07:57.686904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.686814    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\" (UniqueName: \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\") pod \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") "
Oct 30 22:07:57.687089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.686904    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\") pod \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\" (UID: \"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7\") "
Oct 30 22:07:57.687361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.687328    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume" pod "ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" (UID: "ed8b17c3-0612-46e1-886e-ccbd1e7e05c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:57.689427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.689395    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87" pod "ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" (UID: "ed8b17c3-0612-46e1-886e-ccbd1e7e05c7"). InnerVolumeSpecName "kube-api-access-d9k87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:57.713663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.713636    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:07:57.788387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.788349    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\" (UniqueName: \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\") pod \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") "
Oct 30 22:07:57.788598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.788538    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\") pod \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\" (UID: \"094e077c-f7ee-4f22-ab3f-6125ca7fb61f\") "
Oct 30 22:07:57.788900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.788867    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume" pod "094e077c-f7ee-4f22-ab3f-6125ca7fb61f" (UID: "094e077c-f7ee-4f22-ab3f-6125ca7fb61f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:07:57.789014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.788933    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9k87\" (UniqueName: \"kubernetes.io/projected/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-kube-api-access-d9k87\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:57.789014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.788947    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:57.790509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.790488    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66" pod "094e077c-f7ee-4f22-ab3f-6125ca7fb61f" (UID: "094e077c-f7ee-4f22-ab3f-6125ca7fb61f"). InnerVolumeSpecName "kube-api-access-nmb66". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:07:57.890346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.890309    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:57.890346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.890340    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmb66\" (UniqueName: \"kubernetes.io/projected/094e077c-f7ee-4f22-ab3f-6125ca7fb61f-kube-api-access-nmb66\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:07:57.979738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.979651    4882 scope.go:122] "RemoveContainer" containerID="492115f2a2ae8078f7165e3810e6ef3f295f0335362f497f394cd142fd985bba"
Oct 30 22:07:57.985915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.985888    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"]
Oct 30 22:07:57.986487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986474    4882 cpu_manager.go:414] "containerMap: removing container" podUID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" containerName="payload-pre-processing"
Oct 30 22:07:57.986530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986489    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" containerName="payload-pre-processing"
Oct 30 22:07:57.986530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986502    4882 cpu_manager.go:414] "containerMap: removing container" podUID="19ac61fa-88e5-4c36-b26f-73fa556f5516" containerName="revoke-keys"
Oct 30 22:07:57.986530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986507    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19ac61fa-88e5-4c36-b26f-73fa556f5516" containerName="revoke-keys"
Oct 30 22:07:57.986530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986530    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" containerName="payload-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986535    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" containerName="payload-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986543    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3030a906-5209-413d-b120-c919c53ab069" containerName="payload-pre-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986549    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3030a906-5209-413d-b120-c919c53ab069" containerName="payload-pre-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986575    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" containerName="payload-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986580    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" containerName="payload-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986608    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9cdd56c6-601c-4b2a-812f-880713e7622e" containerName="payload-pre-processing"
Oct 30 22:07:57.986683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986613    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9cdd56c6-601c-4b2a-812f-880713e7622e" containerName="payload-pre-processing"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986700    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" containerName="payload-processing"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986710    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9cdd56c6-601c-4b2a-812f-880713e7622e" containerName="payload-pre-processing"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986719    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" containerName="payload-pre-processing"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986726    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="19ac61fa-88e5-4c36-b26f-73fa556f5516" containerName="revoke-keys"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986734    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfe3de7-9fc9-4d09-a4c3-99abaa8b6452" containerName="payload-processing"
Oct 30 22:07:57.986888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.986738    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3030a906-5209-413d-b120-c919c53ab069" containerName="payload-pre-processing"
Oct 30 22:07:57.990391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:57.990370    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.022006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.021964    4882 generic.go:356] "Generic (PLEG): container finished" podID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" containerID="83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f" exitCode=0
Oct 30 22:07:58.022117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.022008    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"
Oct 30 22:07:58.022117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.022025    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" event={"ID":"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7","Type":"ContainerDied","Data":"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"}
Oct 30 22:07:58.022117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.022081    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q" event={"ID":"ed8b17c3-0612-46e1-886e-ccbd1e7e05c7","Type":"ContainerDied","Data":"f597085e19bdd4986b0efa08ae17365ff7ccb9cfa39cd85f28f62ca3e6feba35"}
Oct 30 22:07:58.022117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.022113    4882 scope.go:122] "RemoveContainer" containerID="83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"
Oct 30 22:07:58.026664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.026639    4882 generic.go:356] "Generic (PLEG): container finished" podID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" containerID="9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd" exitCode=0
Oct 30 22:07:58.026781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.026702    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"
Oct 30 22:07:58.026781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.026725    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" event={"ID":"094e077c-f7ee-4f22-ab3f-6125ca7fb61f","Type":"ContainerDied","Data":"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"}
Oct 30 22:07:58.026889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.026782    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j" event={"ID":"094e077c-f7ee-4f22-ab3f-6125ca7fb61f","Type":"ContainerDied","Data":"9318b1020dec03833efc984f59b5a5a37a526c78e19c614825f79e6fa875f516"}
Oct 30 22:07:58.037635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.037600    4882 scope.go:122] "RemoveContainer" containerID="83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"
Oct 30 22:07:58.038008 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:58.037980    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f\": container with ID starting with 83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f not found: ID does not exist" containerID="83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"
Oct 30 22:07:58.038118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.038020    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f"} err="failed to get container status \"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f\": rpc error: code = NotFound desc = could not find container \"83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f\": container with ID starting with 83bbd99ec24a52f8048310426e14953a1e0aecf3043ce81ed69e2112ea27ed9f not found: ID does not exist"
Oct 30 22:07:58.038118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.038046    4882 scope.go:122] "RemoveContainer" containerID="9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"
Oct 30 22:07:58.043482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.043457    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:07:58.043629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.043529    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:07:58.044257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.044230    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"]
Oct 30 22:07:58.050243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.050216    4882 scope.go:122] "RemoveContainer" containerID="9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"
Oct 30 22:07:58.050676 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:07:58.050652    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd\": container with ID starting with 9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd not found: ID does not exist" containerID="9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"
Oct 30 22:07:58.050768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.050687    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd"} err="failed to get container status \"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd\": rpc error: code = NotFound desc = could not find container \"9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd\": container with ID starting with 9198bab05fd68f2feb92c81977e69079b0fe436c6b2d84d6569f10e0545a1dfd not found: ID does not exist"
Oct 30 22:07:58.093286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.093244    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.093547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.093524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.093645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.093567    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwkk5\" (UniqueName: \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.194924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.194888    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.195315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.194936    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwkk5\" (UniqueName: \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.195315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.194978    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.195760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.195738    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.197296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.197277    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.223742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.223708    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:07:58.227669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.227643    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f9b7e7-5dd9975567-lxq6j"]
Oct 30 22:07:58.259368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.259289    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwkk5\" (UniqueName: \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\") pod \"maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.308167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.308133    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:07:58.340362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.340331    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:07:58.344798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.344769    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f9b7e7-7cc759dd8c-tww2q"]
Oct 30 22:07:58.687327 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:07:58.687290    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc017b151_1be0_4b91_a521_b780fc092398.slice/crio-d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0 WatchSource:0}: Error finding container d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0: Status 404 returned error can't find the container with id d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0
Oct 30 22:07:58.690819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.690794    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"]
Oct 30 22:07:58.944576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.944504    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="094e077c-f7ee-4f22-ab3f-6125ca7fb61f" path="/var/lib/kubelet/pods/094e077c-f7ee-4f22-ab3f-6125ca7fb61f/volumes"
Oct 30 22:07:58.945089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:58.945050    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8b17c3-0612-46e1-886e-ccbd1e7e05c7" path="/var/lib/kubelet/pods/ed8b17c3-0612-46e1-886e-ccbd1e7e05c7/volumes"
Oct 30 22:07:59.037876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:07:59.037839    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5" event={"ID":"c017b151-1be0-4b91-a521-b780fc092398","Type":"ContainerStarted","Data":"d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0"}
Oct 30 22:08:00.048550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:00.048470    4882 generic.go:356] "Generic (PLEG): container finished" podID="c017b151-1be0-4b91-a521-b780fc092398" containerID="2a327e98023f865113b148b367781e6a766505701e4332dc5cd30af167d45662" exitCode=0
Oct 30 22:08:00.048550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:00.048513    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5" event={"ID":"c017b151-1be0-4b91-a521-b780fc092398","Type":"ContainerDied","Data":"2a327e98023f865113b148b367781e6a766505701e4332dc5cd30af167d45662"}
Oct 30 22:08:00.529710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:00.529673    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:08:00.529978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:00.529949    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy" containerID="cri-o://1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e" gracePeriod=30
Oct 30 22:08:01.213127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.213100    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:08:01.336404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.336375    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\") pod \"c017b151-1be0-4b91-a521-b780fc092398\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") "
Oct 30 22:08:01.336642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.336413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\" (UniqueName: \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\") pod \"c017b151-1be0-4b91-a521-b780fc092398\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") "
Oct 30 22:08:01.336642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.336623    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\") pod \"c017b151-1be0-4b91-a521-b780fc092398\" (UID: \"c017b151-1be0-4b91-a521-b780fc092398\") "
Oct 30 22:08:01.336836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.336783    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca" pod "c017b151-1be0-4b91-a521-b780fc092398" (UID: "c017b151-1be0-4b91-a521-b780fc092398"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:01.337224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.337196    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c017b151-1be0-4b91-a521-b780fc092398-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:01.339214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.339183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5" pod "c017b151-1be0-4b91-a521-b780fc092398" (UID: "c017b151-1be0-4b91-a521-b780fc092398"). InnerVolumeSpecName "kube-api-access-gwkk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:01.339458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.339419    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca" pod "c017b151-1be0-4b91-a521-b780fc092398" (UID: "c017b151-1be0-4b91-a521-b780fc092398"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:01.438654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.438621    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwkk5\" (UniqueName: \"kubernetes.io/projected/c017b151-1be0-4b91-a521-b780fc092398-kube-api-access-gwkk5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:01.438654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:01.438651    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c017b151-1be0-4b91-a521-b780fc092398-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:02.063387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:02.063358    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"
Oct 30 22:08:02.063566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:02.063361    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5" event={"ID":"c017b151-1be0-4b91-a521-b780fc092398","Type":"ContainerDied","Data":"d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0"}
Oct 30 22:08:02.063566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:02.063475    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1aaf40c2197a66b82a2cd9dc16092129c29127f4eae6c798080c49395af31d0"
Oct 30 22:08:03.136853 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:03.136814    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:08:03.730146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:03.730109    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 22:08:03.730363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:03.730339    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy" containerID="cri-o://4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf" gracePeriod=30
Oct 30 22:08:03.783097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:03.783049    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 22:08:03.783340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:03.783305    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy" containerID="cri-o://126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9" gracePeriod=30
Oct 30 22:08:05.801062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.800997    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:08:05.902146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902112    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902179    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902221    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902242    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902289    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902317    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902464    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:05.902642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902472    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902639    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.902755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:05.902755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902692    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:05.902755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902677    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:05.902755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902681    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\") pod \"7194050e-8c66-4e18-8d67-371a4622757d\" (UID: \"7194050e-8c66-4e18-8d67-371a4622757d\") "
Oct 30 22:08:05.903051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.902976    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:05.903282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.903261    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.903282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.903282    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.903430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.903290    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.903430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.903300    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.903430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.903308    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7194050e-8c66-4e18-8d67-371a4622757d-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:05.904548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.904517    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:05.904925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.904896    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:05.905021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.904958    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:05.905021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:05.904983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28" pod "7194050e-8c66-4e18-8d67-371a4622757d" (UID: "7194050e-8c66-4e18-8d67-371a4622757d"). InnerVolumeSpecName "kube-api-access-rjp28". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:06.005484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.005443    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7194050e-8c66-4e18-8d67-371a4622757d-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:06.005484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.005488    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjp28\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-kube-api-access-rjp28\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:06.005720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.005505    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7194050e-8c66-4e18-8d67-371a4622757d-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:06.005720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.005521    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7194050e-8c66-4e18-8d67-371a4622757d-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:06.102317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.102175    4882 generic.go:356] "Generic (PLEG): container finished" podID="7194050e-8c66-4e18-8d67-371a4622757d" containerID="1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e" exitCode=0
Oct 30 22:08:06.103805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.103482    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" event={"ID":"7194050e-8c66-4e18-8d67-371a4622757d","Type":"ContainerDied","Data":"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"}
Oct 30 22:08:06.103805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.103633    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb" event={"ID":"7194050e-8c66-4e18-8d67-371a4622757d","Type":"ContainerDied","Data":"e7529a9926e79b7e4cf953135c884219a739da906c2d253843f8cf0e9ea28655"}
Oct 30 22:08:06.103805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.103713    4882 scope.go:122] "RemoveContainer" containerID="1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"
Oct 30 22:08:06.104029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.103987    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"
Oct 30 22:08:06.119671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.119649    4882 scope.go:122] "RemoveContainer" containerID="1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"
Oct 30 22:08:06.119984 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:06.119968    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e\": container with ID starting with 1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e not found: ID does not exist" containerID="1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"
Oct 30 22:08:06.120031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.119992    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e"} err="failed to get container status \"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e\": rpc error: code = NotFound desc = could not find container \"1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e\": container with ID starting with 1820fec242d7c01ad465fa609a980dcf9543446b447a9ace58eb17f06a5b3c0e not found: ID does not exist"
Oct 30 22:08:06.273811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.273779    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:08:06.277163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.277135    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-d64gb"]
Oct 30 22:08:06.944258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:06.944180    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7194050e-8c66-4e18-8d67-371a4622757d" path="/var/lib/kubelet/pods/7194050e-8c66-4e18-8d67-371a4622757d/volumes"
Oct 30 22:08:09.074637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.074608    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 22:08:09.091653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.091620    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 22:08:09.126531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.126496    4882 generic.go:356] "Generic (PLEG): container finished" podID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerID="4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf" exitCode=0
Oct 30 22:08:09.126742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.126546    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"
Oct 30 22:08:09.126742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.126573    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" event={"ID":"e617471f-8fa4-4e4e-ad62-f4e9b613bd52","Type":"ContainerDied","Data":"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"}
Oct 30 22:08:09.126742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.126671    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm" event={"ID":"e617471f-8fa4-4e4e-ad62-f4e9b613bd52","Type":"ContainerDied","Data":"a2bb89e70624292504578ac37484155f839e805e6d09535819ac33c0a8e8cab4"}
Oct 30 22:08:09.126742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.126710    4882 scope.go:122] "RemoveContainer" containerID="4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"
Oct 30 22:08:09.128230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.128206    4882 generic.go:356] "Generic (PLEG): container finished" podID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerID="126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9" exitCode=0
Oct 30 22:08:09.128355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.128262    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"
Oct 30 22:08:09.128355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.128287    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" event={"ID":"0c11bb7e-267e-405e-94ba-9b28e6a7bd68","Type":"ContainerDied","Data":"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"}
Oct 30 22:08:09.128355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.128331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v" event={"ID":"0c11bb7e-267e-405e-94ba-9b28e6a7bd68","Type":"ContainerDied","Data":"20c11aef974ab235c02b12febc5e9a819f30a25fa78bad772ade55c2af0f3c9a"}
Oct 30 22:08:09.137043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.137023    4882 scope.go:122] "RemoveContainer" containerID="4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"
Oct 30 22:08:09.137320 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:09.137295    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf\": container with ID starting with 4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf not found: ID does not exist" containerID="4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"
Oct 30 22:08:09.137373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.137328    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf"} err="failed to get container status \"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf\": rpc error: code = NotFound desc = could not find container \"4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf\": container with ID starting with 4f8aa618fb91f0ec7409778c7135e020da92ad36142a896288a0e61e47cdf6bf not found: ID does not exist"
Oct 30 22:08:09.137373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.137344    4882 scope.go:122] "RemoveContainer" containerID="126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"
Oct 30 22:08:09.146777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.146759    4882 scope.go:122] "RemoveContainer" containerID="126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"
Oct 30 22:08:09.147066 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:09.147045    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9\": container with ID starting with 126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9 not found: ID does not exist" containerID="126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"
Oct 30 22:08:09.147159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.147072    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9"} err="failed to get container status \"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9\": rpc error: code = NotFound desc = could not find container \"126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9\": container with ID starting with 126d328f42852ba70b24c5668bfc865f16b77419821be9303c84a6ba4aa476a9 not found: ID does not exist"
Oct 30 22:08:09.246674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246643    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.246865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246696    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.246865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246718    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.246865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246746    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.246865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246787    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.247104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246868    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.247104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246931    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.247104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.246975    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.247268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247020    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.247268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247193    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.247374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247357    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.247438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247363    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.247618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247579    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.247773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.247773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247700    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.247773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.248006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247792    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.248006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247846    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.248006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247890    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\") pod \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\" (UID: \"e617471f-8fa4-4e4e-ad62-f4e9b613bd52\") "
Oct 30 22:08:09.248006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247998    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\") pod \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\" (UID: \"0c11bb7e-267e-405e-94ba-9b28e6a7bd68\") "
Oct 30 22:08:09.248752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.248677    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.248752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.248700    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.250232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.247580    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.248951    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.249110    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.249258    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.249522    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.249864    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:09.250324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.250199    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "kube-api-access-488wz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:09.250648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.250464    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.251707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.251677    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.251926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.251902    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:09.252155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.252132    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "kube-api-access-gd7gv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:09.252155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.252149    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:09.252421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.252395    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:09.252690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.252653    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.252883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.252859    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy" pod "e617471f-8fa4-4e4e-ad62-f4e9b613bd52" (UID: "e617471f-8fa4-4e4e-ad62-f4e9b613bd52"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:09.254089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.254059    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo" pod "0c11bb7e-267e-405e-94ba-9b28e6a7bd68" (UID: "0c11bb7e-267e-405e-94ba-9b28e6a7bd68"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:08:09.350937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350902    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.350937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350931    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.350937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350942    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-488wz\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-kube-api-access-488wz\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350952    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350961    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd7gv\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-kube-api-access-gd7gv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350969    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350978    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350987    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.350997    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351005    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351013    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351021    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351029    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351038    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e617471f-8fa4-4e4e-ad62-f4e9b613bd52-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351046    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.351174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.351054    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0c11bb7e-267e-405e-94ba-9b28e6a7bd68-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:09.491001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.490963    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:08:09.557961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.557928    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 22:08:09.561481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.561446    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-4kb4v"]
Oct 30 22:08:09.596512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.596478    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:08:09.672169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.672083    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 22:08:09.676185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:09.676157    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-s4ndm"]
Oct 30 22:08:10.943119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:10.943084    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" path="/var/lib/kubelet/pods/0c11bb7e-267e-405e-94ba-9b28e6a7bd68/volumes"
Oct 30 22:08:10.943613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:10.943598    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" path="/var/lib/kubelet/pods/e617471f-8fa4-4e4e-ad62-f4e9b613bd52/volumes"
Oct 30 22:08:14.492031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.491988    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" podUID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" containerName="payload-processing" containerID="cri-o://9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0" gracePeriod=25
Oct 30 22:08:14.597122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.597072    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" podUID="16a0c2a8-27de-4e53-bb27-3b91a4906119" containerName="payload-pre-processing" containerID="cri-o://e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47" gracePeriod=25
Oct 30 22:08:14.772508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.772476    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:08:14.844665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.844063    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\") pod \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") "
Oct 30 22:08:14.844665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.844155    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\" (UniqueName: \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\") pod \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\" (UID: \"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37\") "
Oct 30 22:08:14.853962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.853906    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume" pod "b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" (UID: "b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:14.865000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.864967    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft" pod "b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" (UID: "b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37"). InnerVolumeSpecName "kube-api-access-skzft". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:14.885062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.885037    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:08:14.945579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.945552    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\" (UniqueName: \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\") pod \"16a0c2a8-27de-4e53-bb27-3b91a4906119\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") "
Oct 30 22:08:14.945579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.945594    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\") pod \"16a0c2a8-27de-4e53-bb27-3b91a4906119\" (UID: \"16a0c2a8-27de-4e53-bb27-3b91a4906119\") "
Oct 30 22:08:14.946036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.946013    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume" pod "16a0c2a8-27de-4e53-bb27-3b91a4906119" (UID: "16a0c2a8-27de-4e53-bb27-3b91a4906119"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:14.946179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.946157    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16a0c2a8-27de-4e53-bb27-3b91a4906119-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:14.946274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.946182    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:14.946274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.946197    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skzft\" (UniqueName: \"kubernetes.io/projected/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37-kube-api-access-skzft\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:14.947720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:14.947698    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87" pod "16a0c2a8-27de-4e53-bb27-3b91a4906119" (UID: "16a0c2a8-27de-4e53-bb27-3b91a4906119"). InnerVolumeSpecName "kube-api-access-7hx87". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:15.047615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.047516    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hx87\" (UniqueName: \"kubernetes.io/projected/16a0c2a8-27de-4e53-bb27-3b91a4906119-kube-api-access-7hx87\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:15.172019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.171981    4882 generic.go:356] "Generic (PLEG): container finished" podID="16a0c2a8-27de-4e53-bb27-3b91a4906119" containerID="e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47" exitCode=0
Oct 30 22:08:15.172182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.172032    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"
Oct 30 22:08:15.172182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.172069    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" event={"ID":"16a0c2a8-27de-4e53-bb27-3b91a4906119","Type":"ContainerDied","Data":"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"}
Oct 30 22:08:15.172182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.172130    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62" event={"ID":"16a0c2a8-27de-4e53-bb27-3b91a4906119","Type":"ContainerDied","Data":"bf9ab7452aebf991e4ab508366bb98d1bdd6ab5331b94adf7f0d9c1c9c47bbfb"}
Oct 30 22:08:15.172182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.172154    4882 scope.go:122] "RemoveContainer" containerID="e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"
Oct 30 22:08:15.173793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.173768    4882 generic.go:356] "Generic (PLEG): container finished" podID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" containerID="9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0" exitCode=0
Oct 30 22:08:15.173922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.173815    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"
Oct 30 22:08:15.173980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.173913    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" event={"ID":"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37","Type":"ContainerDied","Data":"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"}
Oct 30 22:08:15.174021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.173977    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz" event={"ID":"b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37","Type":"ContainerDied","Data":"5ee66be65b1615d73a07981ba812357d7cab49330c9883545890a183ebd13c59"}
Oct 30 22:08:15.183413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.183397    4882 scope.go:122] "RemoveContainer" containerID="e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"
Oct 30 22:08:15.183730 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:15.183715    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47\": container with ID starting with e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47 not found: ID does not exist" containerID="e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"
Oct 30 22:08:15.183787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.183739    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47"} err="failed to get container status \"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47\": rpc error: code = NotFound desc = could not find container \"e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47\": container with ID starting with e1d085f48e28eba68123fb5ce1a2b5fe3c80f70331288c7ba4bbdf0e1cf84d47 not found: ID does not exist"
Oct 30 22:08:15.183787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.183756    4882 scope.go:122] "RemoveContainer" containerID="9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"
Oct 30 22:08:15.193398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.193374    4882 scope.go:122] "RemoveContainer" containerID="9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"
Oct 30 22:08:15.193680 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:15.193660    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0\": container with ID starting with 9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0 not found: ID does not exist" containerID="9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"
Oct 30 22:08:15.193746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.193691    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0"} err="failed to get container status \"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0\": rpc error: code = NotFound desc = could not find container \"9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0\": container with ID starting with 9351da6722c6c2599969e15c1660d2bfe5bfeed8039b12235ffc22bcd7d54cc0 not found: ID does not exist"
Oct 30 22:08:15.341551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.341509    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:08:15.345319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.345289    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-82b62e-785f85cc9-f6l62"]
Oct 30 22:08:15.464051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.464014    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:08:15.468053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:15.468028    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-82b62e-6c4f666b6c-tbrsz"]
Oct 30 22:08:16.944435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:16.944400    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="16a0c2a8-27de-4e53-bb27-3b91a4906119" path="/var/lib/kubelet/pods/16a0c2a8-27de-4e53-bb27-3b91a4906119/volumes"
Oct 30 22:08:16.944900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:16.944885    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" path="/var/lib/kubelet/pods/b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37/volumes"
Oct 30 22:08:17.749517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.749468    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"]
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751037    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c017b151-1be0-4b91-a521-b780fc092398" containerName="revoke-keys"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751065    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c017b151-1be0-4b91-a521-b780fc092398" containerName="revoke-keys"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751100    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751109    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751138    4882 cpu_manager.go:414] "containerMap: removing container" podUID="16a0c2a8-27de-4e53-bb27-3b91a4906119" containerName="payload-pre-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751147    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="16a0c2a8-27de-4e53-bb27-3b91a4906119" containerName="payload-pre-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751163    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751170    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751187    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751194    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751241    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" containerName="payload-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751248    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" containerName="payload-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751409    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="16a0c2a8-27de-4e53-bb27-3b91a4906119" containerName="payload-pre-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751425    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e617471f-8fa4-4e4e-ad62-f4e9b613bd52" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751437    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c11bb7e-267e-405e-94ba-9b28e6a7bd68" containerName="istio-proxy"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751451    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b91bbde5-b20c-4f0a-8ae7-e1360a9b8f37" containerName="payload-processing"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751463    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c017b151-1be0-4b91-a521-b780fc092398" containerName="revoke-keys"
Oct 30 22:08:17.752054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.751475    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7194050e-8c66-4e18-8d67-371a4622757d" containerName="istio-proxy"
Oct 30 22:08:17.757212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.757189    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.758075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.758046    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"]
Oct 30 22:08:17.810294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.810265    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:08:17.810294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.810281    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:08:17.885140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.885109    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.885361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.885291    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-565mv\" (UniqueName: \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.885423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.885403    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.987078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.987032    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-565mv\" (UniqueName: \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.987628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.987115    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.987628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.987287    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.987851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.987834    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:17.989730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:17.989711    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:18.049525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:18.049442    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-565mv\" (UniqueName: \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\") pod \"maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:18.077413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:18.077379    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:18.454464 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:08:18.454434    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ff58bd3_ba1d_4916_90a3_5f212efd8d64.slice/crio-55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a WatchSource:0}: Error finding container 55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a: Status 404 returned error can't find the container with id 55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a
Oct 30 22:08:18.457202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:18.457177    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"]
Oct 30 22:08:19.205185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:19.205137    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n" event={"ID":"2ff58bd3-ba1d-4916-90a3-5f212efd8d64","Type":"ContainerStarted","Data":"8418e6d951d0bf275db884977f00d1cf1d2aa79e211111f968361a2a078e1249"}
Oct 30 22:08:19.205679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:19.205196    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n" event={"ID":"2ff58bd3-ba1d-4916-90a3-5f212efd8d64","Type":"ContainerStarted","Data":"55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a"}
Oct 30 22:08:19.327072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:19.327005    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n" podStartSLOduration=2.326971422 podStartE2EDuration="2.326971422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:08:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:08:19.324192973 +0000 UTC m=+1822.497517401" watchObservedRunningTime="2026-10-30 22:08:19.326971422 +0000 UTC m=+1822.500295893"
Oct 30 22:08:20.213825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:20.213787    4882 generic.go:356] "Generic (PLEG): container finished" podID="2ff58bd3-ba1d-4916-90a3-5f212efd8d64" containerID="8418e6d951d0bf275db884977f00d1cf1d2aa79e211111f968361a2a078e1249" exitCode=0
Oct 30 22:08:20.214220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:20.213871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n" event={"ID":"2ff58bd3-ba1d-4916-90a3-5f212efd8d64","Type":"ContainerDied","Data":"8418e6d951d0bf275db884977f00d1cf1d2aa79e211111f968361a2a078e1249"}
Oct 30 22:08:20.810268 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:20.810231    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:08:21.146775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.146740    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"]
Oct 30 22:08:21.149951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.149926    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-4c260505-k4nvt"]
Oct 30 22:08:21.386022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.385992    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:21.536285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.536183    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\") pod \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") "
Oct 30 22:08:21.536480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.536461    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\") pod \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") "
Oct 30 22:08:21.536540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.536499    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\" (UniqueName: \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\") pod \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\" (UID: \"2ff58bd3-ba1d-4916-90a3-5f212efd8d64\") "
Oct 30 22:08:21.536581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.536537    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca" pod "2ff58bd3-ba1d-4916-90a3-5f212efd8d64" (UID: "2ff58bd3-ba1d-4916-90a3-5f212efd8d64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:21.539626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.536984    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:21.539626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.538722    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca" pod "2ff58bd3-ba1d-4916-90a3-5f212efd8d64" (UID: "2ff58bd3-ba1d-4916-90a3-5f212efd8d64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:21.539626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.539070    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv" pod "2ff58bd3-ba1d-4916-90a3-5f212efd8d64" (UID: "2ff58bd3-ba1d-4916-90a3-5f212efd8d64"). InnerVolumeSpecName "kube-api-access-565mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:21.638097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.638060    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:21.638097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:21.638093    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-565mv\" (UniqueName: \"kubernetes.io/projected/2ff58bd3-ba1d-4916-90a3-5f212efd8d64-kube-api-access-565mv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:22.229464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:22.229437    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"
Oct 30 22:08:22.229697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:22.229470    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n" event={"ID":"2ff58bd3-ba1d-4916-90a3-5f212efd8d64","Type":"ContainerDied","Data":"55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a"}
Oct 30 22:08:22.229697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:22.229515    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55c91dbe70fbaf5117f5179b30fd24be43c8d6b4516086720dfe3a3910e87e2a"
Oct 30 22:08:22.943312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:22.943271    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3b4d8689-eb18-4c55-b13a-2d7a4199e309" path="/var/lib/kubelet/pods/3b4d8689-eb18-4c55-b13a-2d7a4199e309/volumes"
Oct 30 22:08:26.511015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:26.510985    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:08:29.189056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:29.189020    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:08:29.301785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:29.301743    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:08:33.261485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:33.261448    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:08:34.276034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.275938    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" podUID="c4e5d01f-db42-47b9-803d-6240e5a10487" containerName="payload-processing" containerID="cri-o://2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7" gracePeriod=25
Oct 30 22:08:34.303210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.303150    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" podUID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" containerName="payload-pre-processing" containerID="cri-o://8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2" gracePeriod=25
Oct 30 22:08:34.606643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.606615    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:08:34.631426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.631392    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:08:34.717254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717219    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\") pod \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") "
Oct 30 22:08:34.717463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717279    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\") pod \"c4e5d01f-db42-47b9-803d-6240e5a10487\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") "
Oct 30 22:08:34.717463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717308    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\" (UniqueName: \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\") pod \"c4e5d01f-db42-47b9-803d-6240e5a10487\" (UID: \"c4e5d01f-db42-47b9-803d-6240e5a10487\") "
Oct 30 22:08:34.717463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717411    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\" (UniqueName: \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\") pod \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\" (UID: \"55b30e2a-fba6-44e0-b8a2-4388a75ae502\") "
Oct 30 22:08:34.717718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717690    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume" pod "55b30e2a-fba6-44e0-b8a2-4388a75ae502" (UID: "55b30e2a-fba6-44e0-b8a2-4388a75ae502"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:34.717801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717776    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume" pod "c4e5d01f-db42-47b9-803d-6240e5a10487" (UID: "c4e5d01f-db42-47b9-803d-6240e5a10487"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:08:34.717973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717957    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55b30e2a-fba6-44e0-b8a2-4388a75ae502-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:34.718013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.717976    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c4e5d01f-db42-47b9-803d-6240e5a10487-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:34.719609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.719572    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v" pod "55b30e2a-fba6-44e0-b8a2-4388a75ae502" (UID: "55b30e2a-fba6-44e0-b8a2-4388a75ae502"). InnerVolumeSpecName "kube-api-access-7nl2v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:34.719666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.719625    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7" pod "c4e5d01f-db42-47b9-803d-6240e5a10487" (UID: "c4e5d01f-db42-47b9-803d-6240e5a10487"). InnerVolumeSpecName "kube-api-access-p82f7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:34.819197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.819116    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7nl2v\" (UniqueName: \"kubernetes.io/projected/55b30e2a-fba6-44e0-b8a2-4388a75ae502-kube-api-access-7nl2v\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:34.819197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:34.819146    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p82f7\" (UniqueName: \"kubernetes.io/projected/c4e5d01f-db42-47b9-803d-6240e5a10487-kube-api-access-p82f7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:35.318113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.318071    4882 generic.go:356] "Generic (PLEG): container finished" podID="c4e5d01f-db42-47b9-803d-6240e5a10487" containerID="2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7" exitCode=0
Oct 30 22:08:35.318574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.318125    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"
Oct 30 22:08:35.318574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.318153    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" event={"ID":"c4e5d01f-db42-47b9-803d-6240e5a10487","Type":"ContainerDied","Data":"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"}
Oct 30 22:08:35.318574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.318219    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj" event={"ID":"c4e5d01f-db42-47b9-803d-6240e5a10487","Type":"ContainerDied","Data":"eb9625251d35b80a5d1f8afc9edcc303db0205605a87b5b3cdb53107c0633f29"}
Oct 30 22:08:35.318574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.318241    4882 scope.go:122] "RemoveContainer" containerID="2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"
Oct 30 22:08:35.320019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.319990    4882 generic.go:356] "Generic (PLEG): container finished" podID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" containerID="8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2" exitCode=0
Oct 30 22:08:35.320145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.320036    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"
Oct 30 22:08:35.320145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.320064    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" event={"ID":"55b30e2a-fba6-44e0-b8a2-4388a75ae502","Type":"ContainerDied","Data":"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"}
Oct 30 22:08:35.320145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.320112    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg" event={"ID":"55b30e2a-fba6-44e0-b8a2-4388a75ae502","Type":"ContainerDied","Data":"1e81ba7185c1d971756ee8e20fe1ba186cba5526f07d43edad5345af721daee0"}
Oct 30 22:08:35.329468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.329447    4882 scope.go:122] "RemoveContainer" containerID="2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"
Oct 30 22:08:35.329762 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:35.329744    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7\": container with ID starting with 2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7 not found: ID does not exist" containerID="2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"
Oct 30 22:08:35.329829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.329771    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7"} err="failed to get container status \"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7\": rpc error: code = NotFound desc = could not find container \"2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7\": container with ID starting with 2e5852518e75a24f098631937f93dd385f19791b6e4be39e0553dd66c44a19d7 not found: ID does not exist"
Oct 30 22:08:35.329829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.329787    4882 scope.go:122] "RemoveContainer" containerID="8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"
Oct 30 22:08:35.339407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.339388    4882 scope.go:122] "RemoveContainer" containerID="8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"
Oct 30 22:08:35.339778 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:35.339746    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2\": container with ID starting with 8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2 not found: ID does not exist" containerID="8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"
Oct 30 22:08:35.339893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.339784    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2"} err="failed to get container status \"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2\": rpc error: code = NotFound desc = could not find container \"8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2\": container with ID starting with 8780a91378e28494de3d90de09bc56329a5197e701f4d9268db0634d34ee75d2 not found: ID does not exist"
Oct 30 22:08:35.487640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.487608    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:08:35.537336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.537297    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-4cb675-57b69cd74c-tnncg"]
Oct 30 22:08:35.602574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.602539    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:08:35.607048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:35.607018    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-4cb675-5c9777bf65-nntcj"]
Oct 30 22:08:36.943200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:36.943169    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" path="/var/lib/kubelet/pods/55b30e2a-fba6-44e0-b8a2-4388a75ae502/volumes"
Oct 30 22:08:36.943679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:36.943660    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4e5d01f-db42-47b9-803d-6240e5a10487" path="/var/lib/kubelet/pods/c4e5d01f-db42-47b9-803d-6240e5a10487/volumes"
Oct 30 22:08:41.532387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.532340    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" podUID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" containerName="main" containerID="cri-o://b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec" gracePeriod=45
Oct 30 22:08:41.794828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.794759    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:08:41.902827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.902794    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.902856    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.902884    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\" (UniqueName: \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.902914    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903002    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\" (UniqueName: \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903023    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\") pod \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\" (UID: \"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8\") "
Oct 30 22:08:41.903256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903107    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:41.903256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903193    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:41.903449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903412    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:41.903581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903559    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:41.903676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903605    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:41.903676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.903622    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:41.905269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.905241    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:08:41.905269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.905255    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "kube-api-access-4nqs7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:08:41.905418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:41.905290    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs" pod "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" (UID: "903e1a7f-cf27-49c0-97cc-b6d0ab6229e8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:08:42.004961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.004922    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nqs7\" (UniqueName: \"kubernetes.io/projected/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-kube-api-access-4nqs7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:42.004961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.004958    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:42.004961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.004968    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:08:42.381252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.381202    4882 generic.go:356] "Generic (PLEG): container finished" podID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" containerID="b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec" exitCode=0
Oct 30 22:08:42.381436 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.381376    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" event={"ID":"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8","Type":"ContainerDied","Data":"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"}
Oct 30 22:08:42.381436 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.381415    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86" event={"ID":"903e1a7f-cf27-49c0-97cc-b6d0ab6229e8","Type":"ContainerDied","Data":"295399f7d93491a4ea96b667fa587c7ad64a6e2368f12341bd6ad5e1359f5a33"}
Oct 30 22:08:42.381517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.381448    4882 scope.go:122] "RemoveContainer" containerID="b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"
Oct 30 22:08:42.381701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.381686    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"
Oct 30 22:08:42.394080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.394059    4882 scope.go:122] "RemoveContainer" containerID="b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"
Oct 30 22:08:42.394342 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:42.394321    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec\": container with ID starting with b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec not found: ID does not exist" containerID="b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"
Oct 30 22:08:42.394405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.394351    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec"} err="failed to get container status \"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec\": rpc error: code = NotFound desc = could not find container \"b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec\": container with ID starting with b5909873d89794ee8225afb0ec1cf09a0a4eaeb24e1c14fba27f39cbbea17cec not found: ID does not exist"
Oct 30 22:08:42.549768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.549727    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:08:42.599972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.599935    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-4c260505/e2e-lifecycle-model-4c260505-kserve-bb4849bdf-7dp86"]
Oct 30 22:08:42.943951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:42.943910    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" path="/var/lib/kubelet/pods/903e1a7f-cf27-49c0-97cc-b6d0ab6229e8/volumes"
Oct 30 22:08:50.807724 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:08:50.807681    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:08:55.373681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:55.373640    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"]
Oct 30 22:08:55.423628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:55.423580    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f9b7e7-z7cxr"]
Oct 30 22:08:56.943687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:56.943654    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19ac61fa-88e5-4c36-b26f-73fa556f5516" path="/var/lib/kubelet/pods/19ac61fa-88e5-4c36-b26f-73fa556f5516/volumes"
Oct 30 22:08:58.041115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:58.041083    4882 scope.go:122] "RemoveContainer" containerID="1eed5e2dac265aabf51685e9ac1209d803f4001ee19c985e20643ff7fc2455dc"
Oct 30 22:08:58.051718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:58.051690    4882 scope.go:122] "RemoveContainer" containerID="548ac3f865e6a3abc3fccb7d4d803ced89c1fed9bfc82488cf604ac98f623eb8"
Oct 30 22:08:58.061253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:08:58.061228    4882 scope.go:122] "RemoveContainer" containerID="fa980fdcaf7a352e1309d4ae75895c4e5ba886108682f6599b8816f9954971ef"
Oct 30 22:09:03.413040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.413002    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"]
Oct 30 22:09:03.413945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.413923    4882 cpu_manager.go:414] "containerMap: removing container" podUID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" containerName="payload-pre-processing"
Oct 30 22:09:03.414002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.413948    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" containerName="payload-pre-processing"
Oct 30 22:09:03.414002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.413990    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c4e5d01f-db42-47b9-803d-6240e5a10487" containerName="payload-processing"
Oct 30 22:09:03.414070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414001    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4e5d01f-db42-47b9-803d-6240e5a10487" containerName="payload-processing"
Oct 30 22:09:03.414070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414023    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2ff58bd3-ba1d-4916-90a3-5f212efd8d64" containerName="revoke-keys"
Oct 30 22:09:03.414070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414031    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ff58bd3-ba1d-4916-90a3-5f212efd8d64" containerName="revoke-keys"
Oct 30 22:09:03.414070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414049    4882 cpu_manager.go:414] "containerMap: removing container" podUID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" containerName="main"
Oct 30 22:09:03.414070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414056    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" containerName="main"
Oct 30 22:09:03.414235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414204    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b30e2a-fba6-44e0-b8a2-4388a75ae502" containerName="payload-pre-processing"
Oct 30 22:09:03.414235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414224    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4e5d01f-db42-47b9-803d-6240e5a10487" containerName="payload-processing"
Oct 30 22:09:03.414298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414238    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ff58bd3-ba1d-4916-90a3-5f212efd8d64" containerName="revoke-keys"
Oct 30 22:09:03.414298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.414249    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="903e1a7f-cf27-49c0-97cc-b6d0ab6229e8" containerName="main"
Oct 30 22:09:03.418474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.418453    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.467792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.467752    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.467983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.467835    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pv7q\" (UniqueName: \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.468047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.468027    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.471062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.471029    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"]
Oct 30 22:09:03.517124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.517090    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:09:03.517124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.517090    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:03.569144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.569103    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.569342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.569171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pv7q\" (UniqueName: \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.569410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.569351    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.569995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.569968    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.571471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.571444    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.630665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.630627    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pv7q\" (UniqueName: \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\") pod \"maas-api-revoke-keys-e2e-api-b-97f097-j5dl9\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:03.734856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:03.734770    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:04.111518 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:09:04.111487    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc091581_7e0a_4d0b_94f6_5d23d73a73f7.slice/crio-2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df WatchSource:0}: Error finding container 2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df: Status 404 returned error can't find the container with id 2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df
Oct 30 22:09:04.114767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:04.114692    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"]
Oct 30 22:09:04.545692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:04.545660    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9" event={"ID":"cc091581-7e0a-4d0b-94f6-5d23d73a73f7","Type":"ContainerStarted","Data":"b485f9b600503fa665ab13c95507f7c30a251670e0b5b89cc2ebd65951c3f097"}
Oct 30 22:09:04.546135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:04.545710    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9" event={"ID":"cc091581-7e0a-4d0b-94f6-5d23d73a73f7","Type":"ContainerStarted","Data":"2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df"}
Oct 30 22:09:04.668475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:04.668417    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9" podStartSLOduration=1.668402354 podStartE2EDuration="1.668402354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:04.665724173 +0000 UTC m=+1867.839048601" watchObservedRunningTime="2026-10-30 22:09:04.668402354 +0000 UTC m=+1867.841726780"
Oct 30 22:09:05.383928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:05.383894    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:09:05.556944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:05.556899    4882 generic.go:356] "Generic (PLEG): container finished" podID="cc091581-7e0a-4d0b-94f6-5d23d73a73f7" containerID="b485f9b600503fa665ab13c95507f7c30a251670e0b5b89cc2ebd65951c3f097" exitCode=0
Oct 30 22:09:05.557505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:05.557475    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9" event={"ID":"cc091581-7e0a-4d0b-94f6-5d23d73a73f7","Type":"ContainerDied","Data":"b485f9b600503fa665ab13c95507f7c30a251670e0b5b89cc2ebd65951c3f097"}
Oct 30 22:09:06.721137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.721109    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:06.810933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.810894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\") pod \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") "
Oct 30 22:09:06.811112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.811083    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\") pod \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") "
Oct 30 22:09:06.811172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.811156    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\" (UniqueName: \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\") pod \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\" (UID: \"cc091581-7e0a-4d0b-94f6-5d23d73a73f7\") "
Oct 30 22:09:06.811519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.811486    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca" pod "cc091581-7e0a-4d0b-94f6-5d23d73a73f7" (UID: "cc091581-7e0a-4d0b-94f6-5d23d73a73f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:06.811724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.811708    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:06.813059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.813025    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca" pod "cc091581-7e0a-4d0b-94f6-5d23d73a73f7" (UID: "cc091581-7e0a-4d0b-94f6-5d23d73a73f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:06.813255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.813237    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q" pod "cc091581-7e0a-4d0b-94f6-5d23d73a73f7" (UID: "cc091581-7e0a-4d0b-94f6-5d23d73a73f7"). InnerVolumeSpecName "kube-api-access-8pv7q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:06.913045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.913002    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pv7q\" (UniqueName: \"kubernetes.io/projected/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-kube-api-access-8pv7q\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:06.913045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:06.913037    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc091581-7e0a-4d0b-94f6-5d23d73a73f7-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:07.572388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:07.572358    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"
Oct 30 22:09:07.572388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:07.572370    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9" event={"ID":"cc091581-7e0a-4d0b-94f6-5d23d73a73f7","Type":"ContainerDied","Data":"2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df"}
Oct 30 22:09:07.572631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:07.572415    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2f844c6a884bf0b3620adffeb6396c63c6e0cc73719250beb7b2a44c0dfc00df"
Oct 30 22:09:10.849564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.849522    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:09:10.850029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.849855    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" containerID="cri-o://e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94" gracePeriod=30
Oct 30 22:09:10.872417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.872383    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:09:10.872640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.872616    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" containerID="cri-o://a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7" gracePeriod=30
Oct 30 22:09:10.892546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.892505    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:09:10.892785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:10.892758    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" containerID="cri-o://f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a" gracePeriod=30
Oct 30 22:09:11.055360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:11.055328    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"]
Oct 30 22:09:11.058383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:11.058359    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6804b179-97fjx"]
Oct 30 22:09:12.943133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:12.943086    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62650cb7-cda5-4542-b6c7-30edf4954387" path="/var/lib/kubelet/pods/62650cb7-cda5-4542-b6c7-30edf4954387/volumes"
Oct 30 22:09:14.218141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:14.218096    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:09:14.218543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:14.218168    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:14.344597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:14.344557    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:09:14.344790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:14.344640    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:14.913116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:14.913078    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:09:15.022503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:15.022469    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:09:15.444771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:15.444737    4882 patch_prober.go:29] interesting pod/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:09:15.445229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:15.444805    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:09:16.195649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.195619    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:09:16.224856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.224826    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:09:16.228305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.228276    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:09:16.340171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340134    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340192    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340223    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340244    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340283    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340322    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.340380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340346    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340467    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340504    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340522    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340623    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340655    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.340713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340694    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340727    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340772    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340843    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340882    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340917    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.340957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341002    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341035    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.341100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341090    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341105    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341137    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341183    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341206    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341211    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341302    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341374    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341415    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341419    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.341544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341512    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\") pod \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\" (UID: \"5389c44c-105b-47ad-a1d1-b4e4aeb77506\") "
Oct 30 22:09:16.342103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341687    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.342103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341726    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.342103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341732    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.342103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341768    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\") pod \"011c4c48-7879-4f20-9c99-ba72521e2096\" (UID: \"011c4c48-7879-4f20-9c99-ba72521e2096\") "
Oct 30 22:09:16.342103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.341832    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\") pod \"660b77f6-9528-467e-9019-a272dcbca1b0\" (UID: \"660b77f6-9528-467e-9019-a272dcbca1b0\") "
Oct 30 22:09:16.342455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342435    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342459    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342476    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342489    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342503    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342516    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342529    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342543    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.342556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342557    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/011c4c48-7879-4f20-9c99-ba72521e2096-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.343284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342570    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/660b77f6-9528-467e-9019-a272dcbca1b0-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.343284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342596    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.343284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.342805    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.343284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.343145    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:16.343284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.343235    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:16.343580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.343351    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.343580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.343403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.343760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.343735    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "kube-api-access-lxq8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.345623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345350    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:16.345623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345370    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.345623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "kube-api-access-h28rn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.345821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345661    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.345821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345730    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.345954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345857    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.346003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.345970    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:16.346193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.346170    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo" pod "5389c44c-105b-47ad-a1d1-b4e4aeb77506" (UID: "5389c44c-105b-47ad-a1d1-b4e4aeb77506"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:16.346292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.346198    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token" pod "660b77f6-9528-467e-9019-a272dcbca1b0" (UID: "660b77f6-9528-467e-9019-a272dcbca1b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.347220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.347203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk" pod "011c4c48-7879-4f20-9c99-ba72521e2096" (UID: "011c4c48-7879-4f20-9c99-ba72521e2096"). InnerVolumeSpecName "kube-api-access-2r9qk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444323    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444356    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444368    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444381    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/011c4c48-7879-4f20-9c99-ba72521e2096-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444390    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444398    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444408    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h28rn\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-kube-api-access-h28rn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444417    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444430    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5389c44c-105b-47ad-a1d1-b4e4aeb77506-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444441    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444450    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2r9qk\" (UniqueName: \"kubernetes.io/projected/011c4c48-7879-4f20-9c99-ba72521e2096-kube-api-access-2r9qk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444458    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/660b77f6-9528-467e-9019-a272dcbca1b0-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444466    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5389c44c-105b-47ad-a1d1-b4e4aeb77506-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444480    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lxq8b\" (UniqueName: \"kubernetes.io/projected/660b77f6-9528-467e-9019-a272dcbca1b0-kube-api-access-lxq8b\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444492    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/660b77f6-9528-467e-9019-a272dcbca1b0-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.448678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.444507    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/011c4c48-7879-4f20-9c99-ba72521e2096-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:16.630470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.630430    4882 generic.go:356] "Generic (PLEG): container finished" podID="660b77f6-9528-467e-9019-a272dcbca1b0" containerID="f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a" exitCode=0
Oct 30 22:09:16.630663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.630487    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"
Oct 30 22:09:16.630663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.630519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" event={"ID":"660b77f6-9528-467e-9019-a272dcbca1b0","Type":"ContainerDied","Data":"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"}
Oct 30 22:09:16.630663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.630638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv" event={"ID":"660b77f6-9528-467e-9019-a272dcbca1b0","Type":"ContainerDied","Data":"7aa08dcf3b177feaee09106324f6a573b921e1115f9b1029868ba12980d3fdf8"}
Oct 30 22:09:16.630912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.630668    4882 scope.go:122] "RemoveContainer" containerID="f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"
Oct 30 22:09:16.632097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.632072    4882 generic.go:356] "Generic (PLEG): container finished" podID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerID="e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94" exitCode=0
Oct 30 22:09:16.632186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.632123    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"
Oct 30 22:09:16.632186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.632150    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" event={"ID":"5389c44c-105b-47ad-a1d1-b4e4aeb77506","Type":"ContainerDied","Data":"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"}
Oct 30 22:09:16.632292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.632205    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj" event={"ID":"5389c44c-105b-47ad-a1d1-b4e4aeb77506","Type":"ContainerDied","Data":"e4616f2718eb201d6bcb1077fc75da80e5ff8061ded67187f78de7d150a846e1"}
Oct 30 22:09:16.633769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.633741    4882 generic.go:356] "Generic (PLEG): container finished" podID="011c4c48-7879-4f20-9c99-ba72521e2096" containerID="a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7" exitCode=0
Oct 30 22:09:16.633920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.633804    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"
Oct 30 22:09:16.633920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.633800    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" event={"ID":"011c4c48-7879-4f20-9c99-ba72521e2096","Type":"ContainerDied","Data":"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"}
Oct 30 22:09:16.634036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.633948    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5" event={"ID":"011c4c48-7879-4f20-9c99-ba72521e2096","Type":"ContainerDied","Data":"83953eddb9406a0a4312c730eecf95f47ebdcf54f7b168c344627725aee97c58"}
Oct 30 22:09:16.644845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.644712    4882 scope.go:122] "RemoveContainer" containerID="f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"
Oct 30 22:09:16.648988 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:16.648957    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a\": container with ID starting with f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a not found: ID does not exist" containerID="f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"
Oct 30 22:09:16.649096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.648999    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a"} err="failed to get container status \"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a\": rpc error: code = NotFound desc = could not find container \"f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a\": container with ID starting with f28f1ff6e92ab336b3af8b3b73930d116dfdc95e43d23da71dd470a74eafed8a not found: ID does not exist"
Oct 30 22:09:16.649096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.649023    4882 scope.go:122] "RemoveContainer" containerID="e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"
Oct 30 22:09:16.662858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.662366    4882 scope.go:122] "RemoveContainer" containerID="e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"
Oct 30 22:09:16.662991 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:16.662944    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94\": container with ID starting with e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94 not found: ID does not exist" containerID="e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"
Oct 30 22:09:16.663076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.662984    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94"} err="failed to get container status \"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94\": rpc error: code = NotFound desc = could not find container \"e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94\": container with ID starting with e36ea612cb3f1948fd8bcbb64343116c8d5d5fd40638f6333674f41aa74d9b94 not found: ID does not exist"
Oct 30 22:09:16.663076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.663010    4882 scope.go:122] "RemoveContainer" containerID="a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"
Oct 30 22:09:16.674405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.674384    4882 scope.go:122] "RemoveContainer" containerID="a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"
Oct 30 22:09:16.674773 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:16.674742    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7\": container with ID starting with a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7 not found: ID does not exist" containerID="a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"
Oct 30 22:09:16.674848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.674775    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7"} err="failed to get container status \"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7\": rpc error: code = NotFound desc = could not find container \"a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7\": container with ID starting with a2389c542de8fe8e3738c33b1203a200ea1eacea5bf47aec52ef55390f544ff7 not found: ID does not exist"
Oct 30 22:09:16.885352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.885319    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:09:16.898538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.898506    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-c8mgv"]
Oct 30 22:09:16.943389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:16.943357    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" path="/var/lib/kubelet/pods/660b77f6-9528-467e-9019-a272dcbca1b0/volumes"
Oct 30 22:09:17.010374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:17.010336    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:09:17.016530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:17.016496    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-lg8h5"]
Oct 30 22:09:17.134669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:17.134578    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:09:17.153635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:17.153599    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-4c260505-openshift-default-6f77466685-vmqvj"]
Oct 30 22:09:18.942453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:18.942418    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" path="/var/lib/kubelet/pods/011c4c48-7879-4f20-9c99-ba72521e2096/volumes"
Oct 30 22:09:18.942966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:18.942950    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" path="/var/lib/kubelet/pods/5389c44c-105b-47ad-a1d1-b4e4aeb77506/volumes"
Oct 30 22:09:19.914260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:19.914205    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" podUID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" containerName="payload-processing" containerID="cri-o://beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921" gracePeriod=25
Oct 30 22:09:20.024977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.024928    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" podUID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" containerName="payload-pre-processing" containerID="cri-o://66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828" gracePeriod=25
Oct 30 22:09:20.229942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.229845    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:09:20.284640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.284602    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:09:20.285367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285350    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy"
Oct 30 22:09:20.285432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285371    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy"
Oct 30 22:09:20.285432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285398    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" containerName="payload-processing"
Oct 30 22:09:20.285432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285407    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" containerName="payload-processing"
Oct 30 22:09:20.285432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285428    4882 cpu_manager.go:414] "containerMap: removing container" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy"
Oct 30 22:09:20.285559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285436    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy"
Oct 30 22:09:20.285559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285460    4882 cpu_manager.go:414] "containerMap: removing container" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy"
Oct 30 22:09:20.285559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285467    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy"
Oct 30 22:09:20.285559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285509    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cc091581-7e0a-4d0b-94f6-5d23d73a73f7" containerName="revoke-keys"
Oct 30 22:09:20.285559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285515    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc091581-7e0a-4d0b-94f6-5d23d73a73f7" containerName="revoke-keys"
Oct 30 22:09:20.285742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285675    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5389c44c-105b-47ad-a1d1-b4e4aeb77506" containerName="istio-proxy"
Oct 30 22:09:20.285742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285691    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" containerName="payload-processing"
Oct 30 22:09:20.285742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285702    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="011c4c48-7879-4f20-9c99-ba72521e2096" containerName="istio-proxy"
Oct 30 22:09:20.285742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285714    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b77f6-9528-467e-9019-a272dcbca1b0" containerName="istio-proxy"
Oct 30 22:09:20.285742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.285730    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc091581-7e0a-4d0b-94f6-5d23d73a73f7" containerName="revoke-keys"
Oct 30 22:09:20.288426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.288396    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\") pod \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") "
Oct 30 22:09:20.288566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.288548    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\" (UniqueName: \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\") pod \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\" (UID: \"e2e018ba-a061-4838-aa21-ef0c0d1d408d\") "
Oct 30 22:09:20.288874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.288847    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume" pod "e2e018ba-a061-4838-aa21-ef0c0d1d408d" (UID: "e2e018ba-a061-4838-aa21-ef0c0d1d408d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:20.289052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.289033    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e2e018ba-a061-4838-aa21-ef0c0d1d408d-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:20.291960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.291934    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv" pod "e2e018ba-a061-4838-aa21-ef0c0d1d408d" (UID: "e2e018ba-a061-4838-aa21-ef0c0d1d408d"). InnerVolumeSpecName "kube-api-access-cwddv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:20.292510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.292490    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.324685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.324650    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:09:20.351076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.351029    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:09:20.351355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.351338    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-52f191\""
Oct 30 22:09:20.351478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.351458    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-52f191-dockercfg-vr9lc\""
Oct 30 22:09:20.390091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.390057    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\" (UniqueName: \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\") pod \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") "
Oct 30 22:09:20.390296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.390176    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\") pod \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\" (UID: \"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1\") "
Oct 30 22:09:20.390427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.390409    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sdm92\" (UniqueName: \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.390502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.390493    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.390602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.390575    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwddv\" (UniqueName: \"kubernetes.io/projected/e2e018ba-a061-4838-aa21-ef0c0d1d408d-kube-api-access-cwddv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:20.391169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.391131    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume" pod "2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" (UID: "2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:20.393411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.393380    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p" pod "2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" (UID: "2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1"). InnerVolumeSpecName "kube-api-access-v9p5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:20.461688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.461557    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:09:20.462220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.462202    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" containerName="payload-pre-processing"
Oct 30 22:09:20.462297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.462222    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" containerName="payload-pre-processing"
Oct 30 22:09:20.462355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.462345    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" containerName="payload-pre-processing"
Oct 30 22:09:20.466050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.466025    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.491565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.491524    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.492001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.491972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.492219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.492195    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4jwc\" (UniqueName: \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.492329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.492309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sdm92\" (UniqueName: \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.492758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.492516    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v9p5p\" (UniqueName: \"kubernetes.io/projected/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-kube-api-access-v9p5p\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:20.492758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.492541    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:20.494029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.494005    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.522305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.522261    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:09:20.554760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.554719    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sdm92\" (UniqueName: \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\") pod \"payload-processing-e2e-api-a-52f191-fdd756487-t6lgk\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") " pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.594343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.594302    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.594538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.594351    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4jwc\" (UniqueName: \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.595129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.595090    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.615982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.615950    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:20.657141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.657100    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4jwc\" (UniqueName: \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\") pod \"payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.680039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.680000    4882 generic.go:356] "Generic (PLEG): container finished" podID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" containerID="beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921" exitCode=0
Oct 30 22:09:20.680245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.680061    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"
Oct 30 22:09:20.680245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.680116    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" event={"ID":"e2e018ba-a061-4838-aa21-ef0c0d1d408d","Type":"ContainerDied","Data":"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"}
Oct 30 22:09:20.680245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.680206    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg" event={"ID":"e2e018ba-a061-4838-aa21-ef0c0d1d408d","Type":"ContainerDied","Data":"36a205757c27184ad5a8f170efbb1e954bb762c372eff709ec9ea474f5f29c2e"}
Oct 30 22:09:20.680245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.680240    4882 scope.go:122] "RemoveContainer" containerID="beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"
Oct 30 22:09:20.682034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.681732    4882 generic.go:356] "Generic (PLEG): container finished" podID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" containerID="66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828" exitCode=0
Oct 30 22:09:20.682034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.681777    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" event={"ID":"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1","Type":"ContainerDied","Data":"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"}
Oct 30 22:09:20.682034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.681806    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"
Oct 30 22:09:20.682034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.681817    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg" event={"ID":"2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1","Type":"ContainerDied","Data":"9bd5ab276d9c51f4fef4e2c32a35dc0495ce6f09de325ed96fb7b0d15ce2e916"}
Oct 30 22:09:20.692669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.692645    4882 scope.go:122] "RemoveContainer" containerID="beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"
Oct 30 22:09:20.702881 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:20.699055    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921\": container with ID starting with beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921 not found: ID does not exist" containerID="beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"
Oct 30 22:09:20.702881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.699109    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921"} err="failed to get container status \"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921\": rpc error: code = NotFound desc = could not find container \"beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921\": container with ID starting with beddc09ec50168bec58b70363b2255f96f8b1f84314dbb74f859ff9415499921 not found: ID does not exist"
Oct 30 22:09:20.702881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.699135    4882 scope.go:122] "RemoveContainer" containerID="66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"
Oct 30 22:09:20.721601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.714307    4882 scope.go:122] "RemoveContainer" containerID="66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"
Oct 30 22:09:20.721601 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:20.719867    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828\": container with ID starting with 66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828 not found: ID does not exist" containerID="66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"
Oct 30 22:09:20.721601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.719912    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828"} err="failed to get container status \"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828\": rpc error: code = NotFound desc = could not find container \"66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828\": container with ID starting with 66c646a6c3344b91f92b1bcbd3b758855a499a42508b31e6e9be26c8ab98c828 not found: ID does not exist"
Oct 30 22:09:20.783393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.783355    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:20.812552 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:20.812503    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:09:20.860863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.860834    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:09:20.865732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.865705    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-97f097-66f4b57789-tmjdg"]
Oct 30 22:09:20.945397 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.945359    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e018ba-a061-4838-aa21-ef0c0d1d408d" path="/var/lib/kubelet/pods/e2e018ba-a061-4838-aa21-ef0c0d1d408d/volumes"
Oct 30 22:09:20.976019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.975934    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:09:20.978961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:20.978936    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-97f097-569869857f-zthvg"]
Oct 30 22:09:21.020955 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:09:21.020922    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9b26dab_b667_4cb0_8d8b_343e2a098d06.slice/crio-8382451abd7feca6f0e4a8444a4a24ce7d8a2ee232b0877c310131dcc6057ce0 WatchSource:0}: Error finding container 8382451abd7feca6f0e4a8444a4a24ce7d8a2ee232b0877c310131dcc6057ce0: Status 404 returned error can't find the container with id 8382451abd7feca6f0e4a8444a4a24ce7d8a2ee232b0877c310131dcc6057ce0
Oct 30 22:09:21.024839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.024810    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:09:21.163115 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:09:21.163072    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3907bad2_d4c1_4f0a_b98b_343e32c050f3.slice/crio-e29cf8c18927afc78782a238136556a39ded13b354d3902fc945f1ffbf1bc292 WatchSource:0}: Error finding container e29cf8c18927afc78782a238136556a39ded13b354d3902fc945f1ffbf1bc292: Status 404 returned error can't find the container with id e29cf8c18927afc78782a238136556a39ded13b354d3902fc945f1ffbf1bc292
Oct 30 22:09:21.166339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.166305    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:09:21.290438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.290401    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"]
Oct 30 22:09:21.293727 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.293701    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a5f7b9-88tvf"]
Oct 30 22:09:21.689776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.689738    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" event={"ID":"d9b26dab-b667-4cb0-8d8b-343e2a098d06","Type":"ContainerStarted","Data":"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"}
Oct 30 22:09:21.689992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.689815    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" event={"ID":"d9b26dab-b667-4cb0-8d8b-343e2a098d06","Type":"ContainerStarted","Data":"8382451abd7feca6f0e4a8444a4a24ce7d8a2ee232b0877c310131dcc6057ce0"}
Oct 30 22:09:21.689992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.689870    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:21.691537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.691514    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" event={"ID":"3907bad2-d4c1-4f0a-b98b-343e32c050f3","Type":"ContainerStarted","Data":"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"}
Oct 30 22:09:21.691666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.691549    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" event={"ID":"3907bad2-d4c1-4f0a-b98b-343e32c050f3","Type":"ContainerStarted","Data":"e29cf8c18927afc78782a238136556a39ded13b354d3902fc945f1ffbf1bc292"}
Oct 30 22:09:21.691666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.691607    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:21.808253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.808198    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" podStartSLOduration=1.80818324 podStartE2EDuration="1.80818324s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:21.80644627 +0000 UTC m=+1884.979770697" watchObservedRunningTime="2026-10-30 22:09:21.80818324 +0000 UTC m=+1884.981507665"
Oct 30 22:09:21.926197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:21.926133    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" podStartSLOduration=1.926115008 podStartE2EDuration="1.926115008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:09:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:09:21.925306253 +0000 UTC m=+1885.098630677" watchObservedRunningTime="2026-10-30 22:09:21.926115008 +0000 UTC m=+1885.099439437"
Oct 30 22:09:22.513527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:22.513488    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"]
Oct 30 22:09:22.563689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:22.563651    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-4cb675-rcg6n"]
Oct 30 22:09:22.943626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:22.943577    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1" path="/var/lib/kubelet/pods/2b9d4c76-ccc6-4fcf-ba16-7ed6578c1dc1/volumes"
Oct 30 22:09:22.944062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:22.944049    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ff58bd3-ba1d-4916-90a3-5f212efd8d64" path="/var/lib/kubelet/pods/2ff58bd3-ba1d-4916-90a3-5f212efd8d64/volumes"
Oct 30 22:09:22.944433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:22.944421    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f" path="/var/lib/kubelet/pods/fba36ec1-f1e3-4f53-ad9f-27fbe0928e1f/volumes"
Oct 30 22:09:29.018575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:29.018540    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:09:29.019070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:29.018835    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy" containerID="cri-o://453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68" gracePeriod=30
Oct 30 22:09:32.702516 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:32.702479    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:09:32.703050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:32.702632    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:09:33.269949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:33.269913    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"]
Oct 30 22:09:33.276875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:33.276847    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-82b62e-6ldh5"]
Oct 30 22:09:34.294789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.294718    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:09:34.365240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365202    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365258    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365317    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365350    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365377    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365408    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365498    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365538    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.365744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365557    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:34.365744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:34.365744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365675    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\") pod \"8f7a95fd-d284-4220-a0c1-80cff2c18524\" (UID: \"8f7a95fd-d284-4220-a0c1-80cff2c18524\") "
Oct 30 22:09:34.366074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365892    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:34.366137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366099    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:34.366195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.365926    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:34.366449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366423    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.366536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366451    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.366536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366465    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.366536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366479    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8f7a95fd-d284-4220-a0c1-80cff2c18524-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.366536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.366494    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.367839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.367804    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:34.367839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.367825    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:09:34.368096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.368078    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:09:34.368339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.368317    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp" pod "8f7a95fd-d284-4220-a0c1-80cff2c18524" (UID: "8f7a95fd-d284-4220-a0c1-80cff2c18524"). InnerVolumeSpecName "kube-api-access-4vmwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:34.467487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.467401    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.467487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.467438    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.467487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.467453    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8f7a95fd-d284-4220-a0c1-80cff2c18524-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.467487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.467469    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vmwp\" (UniqueName: \"kubernetes.io/projected/8f7a95fd-d284-4220-a0c1-80cff2c18524-kube-api-access-4vmwp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:34.777947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.777891    4882 generic.go:356] "Generic (PLEG): container finished" podID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerID="453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68" exitCode=0
Oct 30 22:09:34.778132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.777957    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"
Oct 30 22:09:34.778132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.777960    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" event={"ID":"8f7a95fd-d284-4220-a0c1-80cff2c18524","Type":"ContainerDied","Data":"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"}
Oct 30 22:09:34.778132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.778023    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s" event={"ID":"8f7a95fd-d284-4220-a0c1-80cff2c18524","Type":"ContainerDied","Data":"367f32393963ea17df14ef5be2f975d895b34a66936bc3344ba73e60bf1b5784"}
Oct 30 22:09:34.778132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.778048    4882 scope.go:122] "RemoveContainer" containerID="453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"
Oct 30 22:09:34.788789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.788769    4882 scope.go:122] "RemoveContainer" containerID="453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"
Oct 30 22:09:34.789099 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:34.789081    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68\": container with ID starting with 453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68 not found: ID does not exist" containerID="453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"
Oct 30 22:09:34.789143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.789109    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68"} err="failed to get container status \"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68\": rpc error: code = NotFound desc = could not find container \"453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68\": container with ID starting with 453bd53f1457cd8e0d7fc849d4ba4b469b6eac0d6362655edf2b80d6aff17a68 not found: ID does not exist"
Oct 30 22:09:34.943701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.943668    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c017b151-1be0-4b91-a521-b780fc092398" path="/var/lib/kubelet/pods/c017b151-1be0-4b91-a521-b780fc092398/volumes"
Oct 30 22:09:34.948062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.948038    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:09:34.998086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:34.998056    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-ffq7s"]
Oct 30 22:09:35.305121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:35.305091    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:09:36.942661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:36.942536    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" path="/var/lib/kubelet/pods/8f7a95fd-d284-4220-a0c1-80cff2c18524/volumes"
Oct 30 22:09:40.948688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:40.948653    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"]
Oct 30 22:09:40.949167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:40.949154    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy"
Oct 30 22:09:40.949204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:40.949167    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy"
Oct 30 22:09:40.949270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:40.949260    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f7a95fd-d284-4220-a0c1-80cff2c18524" containerName="istio-proxy"
Oct 30 22:09:40.953886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:40.953867    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.007715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.007680    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"]
Oct 30 22:09:41.050697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.050664    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2cwm\" (UniqueName: \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.050858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.050729    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.050858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.050819    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.077292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.077253    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:09:41.077473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.077301    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:09:41.151933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.151901    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2cwm\" (UniqueName: \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.152136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.151957    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.152136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.152019    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.152838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.152809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.154481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.154458    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.214467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.214384    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2cwm\" (UniqueName: \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\") pod \"maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.267568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.267533    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:41.641159 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:09:41.641124    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1436b027_dcac_4180_afd1_61e4f7f3517b.slice/crio-52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338 WatchSource:0}: Error finding container 52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338: Status 404 returned error can't find the container with id 52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338
Oct 30 22:09:41.643869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.643845    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"]
Oct 30 22:09:41.828656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:41.828617    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk" event={"ID":"1436b027-dcac-4180-afd1-61e4f7f3517b","Type":"ContainerStarted","Data":"52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338"}
Oct 30 22:09:42.836398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:42.836364    4882 generic.go:356] "Generic (PLEG): container finished" podID="1436b027-dcac-4180-afd1-61e4f7f3517b" containerID="fb201f4365bfd362e58322167227ad6b43536d836d3b643d20cafa125759e22a" exitCode=0
Oct 30 22:09:42.836398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:42.836403    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk" event={"ID":"1436b027-dcac-4180-afd1-61e4f7f3517b","Type":"ContainerDied","Data":"fb201f4365bfd362e58322167227ad6b43536d836d3b643d20cafa125759e22a"}
Oct 30 22:09:43.981132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:43.981104    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:44.086753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.086717    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\" (UniqueName: \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\") pod \"1436b027-dcac-4180-afd1-61e4f7f3517b\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") "
Oct 30 22:09:44.086753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.086752    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\") pod \"1436b027-dcac-4180-afd1-61e4f7f3517b\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") "
Oct 30 22:09:44.087023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.086828    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\") pod \"1436b027-dcac-4180-afd1-61e4f7f3517b\" (UID: \"1436b027-dcac-4180-afd1-61e4f7f3517b\") "
Oct 30 22:09:44.087182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.087143    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca" pod "1436b027-dcac-4180-afd1-61e4f7f3517b" (UID: "1436b027-dcac-4180-afd1-61e4f7f3517b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:09:44.087406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.087301    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.089047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.089023    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca" pod "1436b027-dcac-4180-afd1-61e4f7f3517b" (UID: "1436b027-dcac-4180-afd1-61e4f7f3517b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:09:44.089141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.089024    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm" pod "1436b027-dcac-4180-afd1-61e4f7f3517b" (UID: "1436b027-dcac-4180-afd1-61e4f7f3517b"). InnerVolumeSpecName "kube-api-access-v2cwm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:09:44.188955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.188918    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v2cwm\" (UniqueName: \"kubernetes.io/projected/1436b027-dcac-4180-afd1-61e4f7f3517b-kube-api-access-v2cwm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.188955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.188951    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1436b027-dcac-4180-afd1-61e4f7f3517b-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:09:44.849873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.849834    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"
Oct 30 22:09:44.850065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.849868    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk" event={"ID":"1436b027-dcac-4180-afd1-61e4f7f3517b","Type":"ContainerDied","Data":"52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338"}
Oct 30 22:09:44.850065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:44.849940    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52d4e6c03f4b3bd23791df0430d875e748bbf7ab36ba16b2499683f5884a4338"
Oct 30 22:09:50.810120 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:09:50.810079    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:09:54.928707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:54.928667    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:09:55.036462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:55.036428    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:09:58.146792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:58.146758    4882 scope.go:122] "RemoveContainer" containerID="ed078b730d6dff29a2cecdb2d9986e6f5384d0d8cf35a9eebea1dfdb13beedbc"
Oct 30 22:09:59.929532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:09:59.929497    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" podUID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" containerName="payload-processing" containerID="cri-o://69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12" gracePeriod=25
Oct 30 22:10:00.037622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.037540    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" podUID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" containerName="payload-pre-processing" containerID="cri-o://9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9" gracePeriod=25
Oct 30 22:10:00.209515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.209485    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:10:00.269769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.269740    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\" (UniqueName: \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\") pod \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") "
Oct 30 22:10:00.269984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.269829    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\") pod \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\" (UID: \"c492f48e-31ba-41cf-a2c8-bbfa9e426fda\") "
Oct 30 22:10:00.270330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.270283    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume" pod "c492f48e-31ba-41cf-a2c8-bbfa9e426fda" (UID: "c492f48e-31ba-41cf-a2c8-bbfa9e426fda"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:00.272147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.272119    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d" pod "c492f48e-31ba-41cf-a2c8-bbfa9e426fda" (UID: "c492f48e-31ba-41cf-a2c8-bbfa9e426fda"). InnerVolumeSpecName "kube-api-access-g4q4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:00.299975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.299949    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:10:00.371774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.371739    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\") pod \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") "
Oct 30 22:10:00.371939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.371827    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\" (UniqueName: \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\") pod \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\" (UID: \"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc\") "
Oct 30 22:10:00.372149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.372136    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4q4d\" (UniqueName: \"kubernetes.io/projected/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-kube-api-access-g4q4d\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:00.372193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.372151    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c492f48e-31ba-41cf-a2c8-bbfa9e426fda-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:00.372193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.372174    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume" pod "f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" (UID: "f58d1df3-b26f-4bb0-b2c4-ab67479b82cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:00.373994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.373971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j" pod "f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" (UID: "f58d1df3-b26f-4bb0-b2c4-ab67479b82cc"). InnerVolumeSpecName "kube-api-access-vq68j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:00.473381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.473298    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:00.473381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.473329    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vq68j\" (UniqueName: \"kubernetes.io/projected/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc-kube-api-access-vq68j\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:00.954453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.954395    4882 generic.go:356] "Generic (PLEG): container finished" podID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" containerID="9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9" exitCode=0
Oct 30 22:10:00.954925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.954628    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"
Oct 30 22:10:00.954925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.954627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" event={"ID":"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc","Type":"ContainerDied","Data":"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"}
Oct 30 22:10:00.954925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.954772    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht" event={"ID":"f58d1df3-b26f-4bb0-b2c4-ab67479b82cc","Type":"ContainerDied","Data":"186ee33df49a96ce7732efa36b54e09f502426a5c61c1939243b29199a00e8fe"}
Oct 30 22:10:00.954925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.954818    4882 scope.go:122] "RemoveContainer" containerID="9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"
Oct 30 22:10:00.957796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.957695    4882 generic.go:356] "Generic (PLEG): container finished" podID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" containerID="69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12" exitCode=0
Oct 30 22:10:00.957796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.957777    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"
Oct 30 22:10:00.957955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.957799    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" event={"ID":"c492f48e-31ba-41cf-a2c8-bbfa9e426fda","Type":"ContainerDied","Data":"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"}
Oct 30 22:10:00.957955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.957832    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz" event={"ID":"c492f48e-31ba-41cf-a2c8-bbfa9e426fda","Type":"ContainerDied","Data":"3e971a4a29199e8ecb3113ec05f1f4187d0da6446226766414bad2c96407329d"}
Oct 30 22:10:00.966477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.966454    4882 scope.go:122] "RemoveContainer" containerID="9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"
Oct 30 22:10:00.966902 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:10:00.966883    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9\": container with ID starting with 9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9 not found: ID does not exist" containerID="9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"
Oct 30 22:10:00.966962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.966912    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9"} err="failed to get container status \"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9\": rpc error: code = NotFound desc = could not find container \"9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9\": container with ID starting with 9deab421ca3d23282f49d7493f8ec6a1f1c6678852f2950a52f72e12cfc9afe9 not found: ID does not exist"
Oct 30 22:10:00.966962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.966929    4882 scope.go:122] "RemoveContainer" containerID="69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"
Oct 30 22:10:00.976107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.976080    4882 scope.go:122] "RemoveContainer" containerID="69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"
Oct 30 22:10:00.976385 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:10:00.976366    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12\": container with ID starting with 69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12 not found: ID does not exist" containerID="69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"
Oct 30 22:10:00.976436 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:00.976395    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12"} err="failed to get container status \"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12\": rpc error: code = NotFound desc = could not find container \"69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12\": container with ID starting with 69a2a574d1bae600fc30a96efe0053e4629ba81410e382c8c74d44148b4e4a12 not found: ID does not exist"
Oct 30 22:10:01.121954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:01.121917    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:10:01.171607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:01.171555    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2f6bc1-699575b47f-w8cht"]
Oct 30 22:10:01.285924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:01.285815    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:10:01.289648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:01.289618    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2f6bc1-545476bd69-sg9nz"]
Oct 30 22:10:02.895019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.894969    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"]
Oct 30 22:10:02.895569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895552    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1436b027-dcac-4180-afd1-61e4f7f3517b" containerName="revoke-keys"
Oct 30 22:10:02.895670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895573    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1436b027-dcac-4180-afd1-61e4f7f3517b" containerName="revoke-keys"
Oct 30 22:10:02.895670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895620    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" containerName="payload-processing"
Oct 30 22:10:02.895670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895626    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" containerName="payload-processing"
Oct 30 22:10:02.895670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895638    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" containerName="payload-pre-processing"
Oct 30 22:10:02.895670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895643    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" containerName="payload-pre-processing"
Oct 30 22:10:02.895835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895748    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" containerName="payload-processing"
Oct 30 22:10:02.895835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895758    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" containerName="payload-pre-processing"
Oct 30 22:10:02.895835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.895771    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1436b027-dcac-4180-afd1-61e4f7f3517b" containerName="revoke-keys"
Oct 30 22:10:02.900386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.900366    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:02.943045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.943010    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c492f48e-31ba-41cf-a2c8-bbfa9e426fda" path="/var/lib/kubelet/pods/c492f48e-31ba-41cf-a2c8-bbfa9e426fda/volumes"
Oct 30 22:10:02.943463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.943448    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f58d1df3-b26f-4bb0-b2c4-ab67479b82cc" path="/var/lib/kubelet/pods/f58d1df3-b26f-4bb0-b2c4-ab67479b82cc/volumes"
Oct 30 22:10:02.955354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.955321    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"]
Oct 30 22:10:02.999108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.999063    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:02.999288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.999251    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:02.999334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:02.999288    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ft46g\" (UniqueName: \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.001264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.001245    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:10:03.001387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.001375    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:10:03.100966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.100932    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.101138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.101039    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.101138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.101067    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ft46g\" (UniqueName: \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.101647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.101618    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.103550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.103527    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.171540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.171462    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ft46g\" (UniqueName: \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\") pod \"maas-api-revoke-sub-0b539382-l6cdd\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.214490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.214454    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:03.627337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.627297    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"]
Oct 30 22:10:03.980938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.980889    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd" event={"ID":"29463ad4-cbd4-40aa-aa30-147fa738beba","Type":"ContainerStarted","Data":"d3342c87b050601f0741121e4719f5e2a46530679a1c04f33f32b1fd8ae42e3d"}
Oct 30 22:10:03.981358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:03.980958    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd" event={"ID":"29463ad4-cbd4-40aa-aa30-147fa738beba","Type":"ContainerStarted","Data":"c64534f97ec9d5a59e1f72497d041397d5ccecf2b0b364207c6bca33ba520f6e"}
Oct 30 22:10:04.102095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:04.102025    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd" podStartSLOduration=2.102003574 podStartE2EDuration="2.102003574s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:04.097255317 +0000 UTC m=+1927.270579743" watchObservedRunningTime="2026-10-30 22:10:04.102003574 +0000 UTC m=+1927.275328002"
Oct 30 22:10:04.988311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:04.988278    4882 generic.go:356] "Generic (PLEG): container finished" podID="29463ad4-cbd4-40aa-aa30-147fa738beba" containerID="d3342c87b050601f0741121e4719f5e2a46530679a1c04f33f32b1fd8ae42e3d" exitCode=0
Oct 30 22:10:04.988748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:04.988338    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd" event={"ID":"29463ad4-cbd4-40aa-aa30-147fa738beba","Type":"ContainerDied","Data":"d3342c87b050601f0741121e4719f5e2a46530679a1c04f33f32b1fd8ae42e3d"}
Oct 30 22:10:06.143715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.143686    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:06.235934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.235898    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\" (UniqueName: \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\") pod \"29463ad4-cbd4-40aa-aa30-147fa738beba\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") "
Oct 30 22:10:06.236117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.235997    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\") pod \"29463ad4-cbd4-40aa-aa30-147fa738beba\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") "
Oct 30 22:10:06.236117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.236092    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\") pod \"29463ad4-cbd4-40aa-aa30-147fa738beba\" (UID: \"29463ad4-cbd4-40aa-aa30-147fa738beba\") "
Oct 30 22:10:06.236443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.236414    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca" pod "29463ad4-cbd4-40aa-aa30-147fa738beba" (UID: "29463ad4-cbd4-40aa-aa30-147fa738beba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:06.236574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.236503    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:06.238455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.238426    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca" pod "29463ad4-cbd4-40aa-aa30-147fa738beba" (UID: "29463ad4-cbd4-40aa-aa30-147fa738beba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:10:06.238577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.238512    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g" pod "29463ad4-cbd4-40aa-aa30-147fa738beba" (UID: "29463ad4-cbd4-40aa-aa30-147fa738beba"). InnerVolumeSpecName "kube-api-access-ft46g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:06.337285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.337248    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/29463ad4-cbd4-40aa-aa30-147fa738beba-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:06.337285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:06.337281    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ft46g\" (UniqueName: \"kubernetes.io/projected/29463ad4-cbd4-40aa-aa30-147fa738beba-kube-api-access-ft46g\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:07.002058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:07.002026    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"
Oct 30 22:10:07.002058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:07.002052    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd" event={"ID":"29463ad4-cbd4-40aa-aa30-147fa738beba","Type":"ContainerDied","Data":"c64534f97ec9d5a59e1f72497d041397d5ccecf2b0b364207c6bca33ba520f6e"}
Oct 30 22:10:07.002317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:07.002095    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c64534f97ec9d5a59e1f72497d041397d5ccecf2b0b364207c6bca33ba520f6e"
Oct 30 22:10:17.009510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:17.009475    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"]
Oct 30 22:10:17.059925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:17.059890    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-97f097-j5dl9"]
Oct 30 22:10:18.547528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:18.547490    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:10:18.943032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:18.942994    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc091581-7e0a-4d0b-94f6-5d23d73a73f7" path="/var/lib/kubelet/pods/cc091581-7e0a-4d0b-94f6-5d23d73a73f7/volumes"
Oct 30 22:10:20.806706 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:10:20.806668    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:10:21.546517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.546482    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:10:21.546984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.546970    4882 cpu_manager.go:414] "containerMap: removing container" podUID="29463ad4-cbd4-40aa-aa30-147fa738beba" containerName="revoke-keys"
Oct 30 22:10:21.547045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.546985    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29463ad4-cbd4-40aa-aa30-147fa738beba" containerName="revoke-keys"
Oct 30 22:10:21.547097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.547088    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="29463ad4-cbd4-40aa-aa30-147fa738beba" containerName="revoke-keys"
Oct 30 22:10:21.550780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.550755    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.609441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.608985    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:10:21.616694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.616658    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.616893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.616718    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfwnq\" (UniqueName: \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.654452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.654409    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0b8c9\""
Oct 30 22:10:21.654644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.654477    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0b8c9-dockercfg-2fws7\""
Oct 30 22:10:21.717861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.717818    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.718041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.717873    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfwnq\" (UniqueName: \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.718532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.718513    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.780379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.780345    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfwnq\" (UniqueName: \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\") pod \"payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:21.868812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:21.868779    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:22.244443 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:10:22.244410    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97fabdc8_1e5f_498e_b341_327185f2a676.slice/crio-6f4a33d411b92121ecb6210a98585d606abba9dc6616d266d4f71df83aa121b0 WatchSource:0}: Error finding container 6f4a33d411b92121ecb6210a98585d606abba9dc6616d266d4f71df83aa121b0: Status 404 returned error can't find the container with id 6f4a33d411b92121ecb6210a98585d606abba9dc6616d266d4f71df83aa121b0
Oct 30 22:10:22.247089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:22.247062    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:10:23.105024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:23.104988    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" event={"ID":"97fabdc8-1e5f-498e-b341-327185f2a676","Type":"ContainerStarted","Data":"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"}
Oct 30 22:10:23.105480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:23.105045    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" event={"ID":"97fabdc8-1e5f-498e-b341-327185f2a676","Type":"ContainerStarted","Data":"6f4a33d411b92121ecb6210a98585d606abba9dc6616d266d4f71df83aa121b0"}
Oct 30 22:10:23.105480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:23.105075    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:23.225764 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:23.225698    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" podStartSLOduration=2.225679129 podStartE2EDuration="2.225679129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:23.223877818 +0000 UTC m=+1946.397202246" watchObservedRunningTime="2026-10-30 22:10:23.225679129 +0000 UTC m=+1946.399003556"
Oct 30 22:10:32.995572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:32.995529    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 22:10:32.996216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:32.995779    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy" containerID="cri-o://c318244639b61a5a370e76ebc707208f98ad85ed3a5db3ec77d8159f197ee2da" gracePeriod=30
Oct 30 22:10:34.118327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:34.118286    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:10:34.978547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:34.978508    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:10:34.982722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:34.982701    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:34.997188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:34.997152    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:10:35.001113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.001089    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.071898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.071851    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.071898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.071896    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.071998    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072215    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072254    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072325    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072354    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072371    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072387    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072445    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hhrv\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072539    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj4bs\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072666    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072775    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.072897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072877    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.072977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.072956    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.073277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.073245    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.089632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.081011    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:10:35.089632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.088242    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:10:35.174689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174638    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174697    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174748    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174795    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174836    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174893    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174927    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174966    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.174991    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175015    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175040    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hhrv\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175043    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175068    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj4bs\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175212    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175262    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175809    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.175942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175944    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.175992    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176032    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176113    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.176171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.176525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176296    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.176525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176391    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.176525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.176421    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.177889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.177863    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.178023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.177967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.178108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.178087    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.178614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.178554    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.178707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.178612    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.236991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.236887    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.236991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.236953    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.237356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.237336    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj4bs\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\") pod \"maas-default-gateway-openshift-default-7df854bf67-ksglt\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.237427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.237404    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hhrv\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\") pod \"maas-default-gateway-openshift-default-7df854bf67-wlnm5\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") " pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.297017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.296979    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:35.314428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.314397    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:35.726320 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:10:35.726280    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd971a92c_0cb0_42c0_87fe_1257153ba41d.slice/crio-cc7f457cfe8bfa15ff2caecb598af7e7956242a78f104a668603371ab7860e31 WatchSource:0}: Error finding container cc7f457cfe8bfa15ff2caecb598af7e7956242a78f104a668603371ab7860e31: Status 404 returned error can't find the container with id cc7f457cfe8bfa15ff2caecb598af7e7956242a78f104a668603371ab7860e31
Oct 30 22:10:35.729783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.729727    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.729896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.729847    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.729952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.729896    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.730050 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:10:35.730025    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ccb6660_3298_4d45_88c2_067273851619.slice/crio-dcc9e0bc60c496d29397b9bf2d36a9f9002419b4806d083b89a52bc21d17058a WatchSource:0}: Error finding container dcc9e0bc60c496d29397b9bf2d36a9f9002419b4806d083b89a52bc21d17058a: Status 404 returned error can't find the container with id dcc9e0bc60c496d29397b9bf2d36a9f9002419b4806d083b89a52bc21d17058a
Oct 30 22:10:35.730480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.730346    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:10:35.732560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.732527    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.732637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.732624    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.732683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.732666    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:10:35.734072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:35.734049    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:10:36.194306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.194267    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" event={"ID":"4ccb6660-3298-4d45-88c2-067273851619","Type":"ContainerStarted","Data":"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"}
Oct 30 22:10:36.194774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.194334    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" event={"ID":"4ccb6660-3298-4d45-88c2-067273851619","Type":"ContainerStarted","Data":"dcc9e0bc60c496d29397b9bf2d36a9f9002419b4806d083b89a52bc21d17058a"}
Oct 30 22:10:36.196569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.196532    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" event={"ID":"d971a92c-0cb0-42c0-87fe-1257153ba41d","Type":"ContainerStarted","Data":"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"}
Oct 30 22:10:36.196753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.196598    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" event={"ID":"d971a92c-0cb0-42c0-87fe-1257153ba41d","Type":"ContainerStarted","Data":"cc7f457cfe8bfa15ff2caecb598af7e7956242a78f104a668603371ab7860e31"}
Oct 30 22:10:36.297936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.297209    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:36.299680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.299640    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:36.299877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.299708    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:36.318398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.318367    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:36.318789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.318751    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:36.318988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.318964    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:36.474993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.474863    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podStartSLOduration=2.474839286 podStartE2EDuration="2.474839286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:36.319580475 +0000 UTC m=+1959.492904903" watchObservedRunningTime="2026-10-30 22:10:36.474839286 +0000 UTC m=+1959.648163715"
Oct 30 22:10:36.478684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:36.476935    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podStartSLOduration=2.47691984 podStartE2EDuration="2.47691984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:36.460024241 +0000 UTC m=+1959.633348683" watchObservedRunningTime="2026-10-30 22:10:36.47691984 +0000 UTC m=+1959.650244266"
Oct 30 22:10:37.298192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:37.298150    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:37.298629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:37.298221    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:37.315784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:37.315746    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:37.315942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:37.315828    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:38.209909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.209876    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:10:38.212763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.212736    4882 generic.go:356] "Generic (PLEG): container finished" podID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerID="c318244639b61a5a370e76ebc707208f98ad85ed3a5db3ec77d8159f197ee2da" exitCode=0
Oct 30 22:10:38.214060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.214037    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" event={"ID":"747e0db1-77d5-45bc-9a8f-c7f8154fd74a","Type":"ContainerDied","Data":"c318244639b61a5a370e76ebc707208f98ad85ed3a5db3ec77d8159f197ee2da"}
Oct 30 22:10:38.214180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.214119    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.287670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.287555    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-65659d\""
Oct 30 22:10:38.294191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.294155    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:10:38.297651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.297416    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:38.297651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.297456    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:38.318889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.318781    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:38.318889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.318843    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:38.401632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.362003    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-65659d-dockercfg-4smct\""
Oct 30 22:10:38.405069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.405017    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qnmjs\" (UniqueName: \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.406388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.405555    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.468773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.454200    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:10:38.496749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.495493    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 22:10:38.496749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.496307    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:10:38.496749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.496407    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.507737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.507707    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.507867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.507756    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.507908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.507868    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.507995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.507950    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.507992    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508022    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508053    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508126    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508169    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\") pod \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\" (UID: \"747e0db1-77d5-45bc-9a8f-c7f8154fd74a\") "
Oct 30 22:10:38.508396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508360    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq22c\" (UniqueName: \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.508493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508411    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.508644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508621    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qnmjs\" (UniqueName: \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.508726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508663    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.508874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508848    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.509002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.508984    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.509064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.509046    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.509347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.509324    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:38.509922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.509898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.510065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.509919    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.510293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.510246    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:38.510709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.510658    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:38.511238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.511216    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:38.511943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.511913    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd" pod "747e0db1-77d5-45bc-9a8f-c7f8154fd74a" (UID: "747e0db1-77d5-45bc-9a8f-c7f8154fd74a"). InnerVolumeSpecName "kube-api-access-xw6qd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:38.578286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.578212    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qnmjs\" (UniqueName: \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\") pod \"payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.596712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.596683    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:38.609721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.609412    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq22c\" (UniqueName: \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.609721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.609607    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.610413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610388    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.610547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610531    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610549    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610558    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610578    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610608    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610618    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610634    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610643    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.610945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.610659    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xw6qd\" (UniqueName: \"kubernetes.io/projected/747e0db1-77d5-45bc-9a8f-c7f8154fd74a-kube-api-access-xw6qd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:38.683850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.683655    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq22c\" (UniqueName: \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\") pod \"payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:38.811628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:38.811561    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:39.218668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.195010    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:10:39.300239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.300205    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:39.310001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.300423    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:39.319083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.318972    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:39.319544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.319084    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:39.319866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.319801    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"
Oct 30 22:10:39.319866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.319662    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h" event={"ID":"747e0db1-77d5-45bc-9a8f-c7f8154fd74a","Type":"ContainerDied","Data":"cb128e91db541f759b352fa20a0b137658f78d51f735cc88902c6ef8f632de27"}
Oct 30 22:10:39.319999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.319979    4882 scope.go:122] "RemoveContainer" containerID="c318244639b61a5a370e76ebc707208f98ad85ed3a5db3ec77d8159f197ee2da"
Oct 30 22:10:39.388099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.388056    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:10:39.522756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.522710    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 22:10:39.605733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:39.605696    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-b495h"]
Oct 30 22:10:40.297952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.297907    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:40.298173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.297959    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:40.315795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.315754    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:40.316009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.315804    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:40.337613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.336809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" event={"ID":"52ab1f9d-bb5d-46de-a623-a79b121506d6","Type":"ContainerStarted","Data":"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"}
Oct 30 22:10:40.337613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.336883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" event={"ID":"52ab1f9d-bb5d-46de-a623-a79b121506d6","Type":"ContainerStarted","Data":"897ec8d9ed37961e3d134f28a60888ad9779a803d3bd7571397d7975f92ce5b8"}
Oct 30 22:10:40.338217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.337945    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:40.341610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.341176    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" event={"ID":"6b841170-148b-4fc4-a0fa-999d491259c4","Type":"ContainerStarted","Data":"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"}
Oct 30 22:10:40.341610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.341245    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" event={"ID":"6b841170-148b-4fc4-a0fa-999d491259c4","Type":"ContainerStarted","Data":"98cc3eb2733d92422f4cff77f37b5f89f191606f1b5027557cf7dc61eb4a6f0b"}
Oct 30 22:10:40.345610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.342022    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:40.462612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.461864    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" podStartSLOduration=2.461833065 podStartE2EDuration="2.461833065s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:40.458013139 +0000 UTC m=+1963.631337568" watchObservedRunningTime="2026-10-30 22:10:40.461833065 +0000 UTC m=+1963.635157493"
Oct 30 22:10:40.579676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.578882    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" podStartSLOduration=2.578859021 podStartE2EDuration="2.578859021s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:40.577004506 +0000 UTC m=+1963.750328934" watchObservedRunningTime="2026-10-30 22:10:40.578859021 +0000 UTC m=+1963.752183450"
Oct 30 22:10:40.964613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:40.964183    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" path="/var/lib/kubelet/pods/747e0db1-77d5-45bc-9a8f-c7f8154fd74a/volumes"
Oct 30 22:10:41.298396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:41.298306    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:41.298396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:41.298366    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:41.315346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:41.315315    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:41.315485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:41.315365    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:42.297581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:42.297545    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:42.298175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:42.297617    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:42.315071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:42.315029    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:42.315260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:42.315102    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:43.297378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:43.297338    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksglt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused" start-of-body=
Oct 30 22:10:43.297677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:43.297407    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.143:15021/healthz/ready\": dial tcp 10.132.0.143:15021: connect: connection refused"
Oct 30 22:10:43.315772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:43.315731    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-wlnm5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused" start-of-body=
Oct 30 22:10:43.315938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:43.315805    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.144:15021/healthz/ready\": dial tcp 10.132.0.144:15021: connect: connection refused"
Oct 30 22:10:44.765548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.764582    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:44.765548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.765221    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:44.766703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.766679    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:10:44.791726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.791688    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:44.792138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.792104    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:44.793155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:44.793132    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:10:46.797958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:46.797926    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 22:10:46.798442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:46.798139    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" containerID="cri-o://cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b" gracePeriod=30
Oct 30 22:10:48.567626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.567569    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:10:48.568188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.568174    4882 cpu_manager.go:414] "containerMap: removing container" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy"
Oct 30 22:10:48.568237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.568189    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy"
Oct 30 22:10:48.568280 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.568271    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="747e0db1-77d5-45bc-9a8f-c7f8154fd74a" containerName="istio-proxy"
Oct 30 22:10:48.572965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.572945    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.684991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.684959    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-a3aa65\""
Oct 30 22:10:48.685374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.685347    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-a3aa65-dockercfg-q4bh9\""
Oct 30 22:10:48.685374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.685355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.685610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.685411    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnnn\" (UniqueName: \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.688965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.688884    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:10:48.692055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.692027    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:10:48.697395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.697371    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.742076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.742045    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:10:48.786471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.786439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rswkm\" (UniqueName: \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.786667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.786558    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.786667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.786642    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnnn\" (UniqueName: \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.786751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.786685    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.787287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.787270    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.858872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.858842    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnnn\" (UniqueName: \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\") pod \"payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.887078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.887042    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:48.887444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.887420    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.887555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.887544    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rswkm\" (UniqueName: \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.888249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.888231    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:48.950305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:48.950270    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rswkm\" (UniqueName: \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\") pod \"payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") " pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:49.014381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.014338    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:49.233356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.233283    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:10:49.233356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.233331    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:10:49.262152 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:10:49.262118    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd95da3c_42a3_4d35_9cb4_2e03c0f6c01a.slice/crio-c002cfe977b3fcecdb3fa30dc9086176eb05c6b7fc8657d405b4ce25aceff643 WatchSource:0}: Error finding container c002cfe977b3fcecdb3fa30dc9086176eb05c6b7fc8657d405b4ce25aceff643: Status 404 returned error can't find the container with id c002cfe977b3fcecdb3fa30dc9086176eb05c6b7fc8657d405b4ce25aceff643
Oct 30 22:10:49.266049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.266022    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:10:49.401530 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:10:49.401497    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaf4a5f9_1e1f_49d2_b529_be47062ece43.slice/crio-4e212d11900acdd1aafc945c9fdb9540d17d6e00de43c45b81b8482dddc3cbeb WatchSource:0}: Error finding container 4e212d11900acdd1aafc945c9fdb9540d17d6e00de43c45b81b8482dddc3cbeb: Status 404 returned error can't find the container with id 4e212d11900acdd1aafc945c9fdb9540d17d6e00de43c45b81b8482dddc3cbeb
Oct 30 22:10:49.404442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.404410    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:10:49.441719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.441686    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" event={"ID":"aaf4a5f9-1e1f-49d2-b529-be47062ece43","Type":"ContainerStarted","Data":"4e212d11900acdd1aafc945c9fdb9540d17d6e00de43c45b81b8482dddc3cbeb"}
Oct 30 22:10:49.443653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.443620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" event={"ID":"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a","Type":"ContainerStarted","Data":"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"}
Oct 30 22:10:49.443774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.443678    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" event={"ID":"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a","Type":"ContainerStarted","Data":"c002cfe977b3fcecdb3fa30dc9086176eb05c6b7fc8657d405b4ce25aceff643"}
Oct 30 22:10:49.443832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.443812    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:10:49.563129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:49.563014    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" podStartSLOduration=1.563000164 podStartE2EDuration="1.563000164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:49.561562192 +0000 UTC m=+1972.734886618" watchObservedRunningTime="2026-10-30 22:10:49.563000164 +0000 UTC m=+1972.736324632"
Oct 30 22:10:50.452039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:50.451994    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" event={"ID":"aaf4a5f9-1e1f-49d2-b529-be47062ece43","Type":"ContainerStarted","Data":"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"}
Oct 30 22:10:50.452504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:50.452263    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:10:50.570470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:50.570413    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" podStartSLOduration=2.570392253 podStartE2EDuration="2.570392253s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:10:50.568577527 +0000 UTC m=+1973.741901965" watchObservedRunningTime="2026-10-30 22:10:50.570392253 +0000 UTC m=+1973.743716682"
Oct 30 22:10:50.812785 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:10:50.812691    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:10:52.076625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.076553    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 22:10:52.125508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125470    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125522    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125560    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125678    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125700    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125774    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125862    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.125967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.125944    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\") pod \"10c61a5f-f9f1-448f-a5f1-f371e0444500\" (UID: \"10c61a5f-f9f1-448f-a5f1-f371e0444500\") "
Oct 30 22:10:52.126177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126005    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:52.126177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126051    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:52.126177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126069    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:52.126177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126110    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:10:52.126401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126240    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:52.126641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126569    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10c61a5f-f9f1-448f-a5f1-f371e0444500-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.126641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126625    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.126796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126642    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.126796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126657    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.126796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.126671    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.128057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.128020    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:10:52.128159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.128051    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "kube-api-access-cjl45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:52.128159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.128087    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:10:52.128235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.128183    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token" pod "10c61a5f-f9f1-448f-a5f1-f371e0444500" (UID: "10c61a5f-f9f1-448f-a5f1-f371e0444500"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:10:52.228084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.228047    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.228084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.228081    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.228296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.228094    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10c61a5f-f9f1-448f-a5f1-f371e0444500-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.228296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.228104    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cjl45\" (UniqueName: \"kubernetes.io/projected/10c61a5f-f9f1-448f-a5f1-f371e0444500-kube-api-access-cjl45\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:10:52.380768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.380739    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:10:52.380921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.380805    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:10:52.466074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.466041    4882 generic.go:356] "Generic (PLEG): container finished" podID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerID="cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b" exitCode=0
Oct 30 22:10:52.466252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.466092    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"
Oct 30 22:10:52.466252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.466127    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" event={"ID":"10c61a5f-f9f1-448f-a5f1-f371e0444500","Type":"ContainerDied","Data":"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"}
Oct 30 22:10:52.466252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.466185    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s" event={"ID":"10c61a5f-f9f1-448f-a5f1-f371e0444500","Type":"ContainerDied","Data":"4169f17d6cbe80bf00bfb565c980e20d731d2dc2fbc16a923c947ab8649808a0"}
Oct 30 22:10:52.466252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.466200    4882 scope.go:122] "RemoveContainer" containerID="cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"
Oct 30 22:10:52.481108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.481081    4882 scope.go:122] "RemoveContainer" containerID="cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"
Oct 30 22:10:52.481444 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:10:52.481424    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b\": container with ID starting with cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b not found: ID does not exist" containerID="cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"
Oct 30 22:10:52.481519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.481457    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b"} err="failed to get container status \"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b\": rpc error: code = NotFound desc = could not find container \"cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b\": container with ID starting with cee5dbb746d9b6389078a2f0f507e879cd6ef41e94d908730fcc7f350fdfbd1b not found: ID does not exist"
Oct 30 22:10:52.734072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.733984    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 22:10:52.737071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.737041    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-pbk5s"]
Oct 30 22:10:52.943560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:52.943529    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" path="/var/lib/kubelet/pods/10c61a5f-f9f1-448f-a5f1-f371e0444500/volumes"
Oct 30 22:10:58.201556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:58.201518    4882 scope.go:122] "RemoveContainer" containerID="d292fdce8c20b2627b6f9a98c0fe856be930d6035ed2b88e05bd3a979d347279"
Oct 30 22:10:59.682416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:59.682378    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"]
Oct 30 22:10:59.732855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:10:59.732821    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-2f6bc1-hd8fk"]
Oct 30 22:11:00.455458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:00.455431    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:11:00.944055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:00.944013    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1436b027-dcac-4180-afd1-61e4f7f3517b" path="/var/lib/kubelet/pods/1436b027-dcac-4180-afd1-61e4f7f3517b/volumes"
Oct 30 22:11:01.462772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:01.462737    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:11:18.788132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:18.788050    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:11:20.809349 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:11:20.809306    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:11:29.118337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.118301    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:11:29.118913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.118897    4882 cpu_manager.go:414] "containerMap: removing container" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy"
Oct 30 22:11:29.118959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.118915    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy"
Oct 30 22:11:29.119038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.119022    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="10c61a5f-f9f1-448f-a5f1-f371e0444500" containerName="istio-proxy"
Oct 30 22:11:29.122956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.122937    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.178362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.178320    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4257d\""
Oct 30 22:11:29.233929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.233731    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:11:29.233929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.233886    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4257d-dockercfg-pxfhb\""
Oct 30 22:11:29.243127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.243093    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:11:29.247221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.247194    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.280627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.280572    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lv9f\" (UniqueName: \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.280808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.280750    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.309758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.309724    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:11:29.382073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.381995    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.382073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.382038    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lv9f\" (UniqueName: \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.382264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.382149    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.382264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.382245    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4b985\" (UniqueName: \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.382858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.382842    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.444193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.444155    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lv9f\" (UniqueName: \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.483431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.483393    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.483633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.483498    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4b985\" (UniqueName: \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.484164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.484143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.550898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.550864    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4b985\" (UniqueName: \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\") pod \"payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.563744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.563714    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:29.737530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.737440    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:29.945947 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:11:29.945899    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0057df5_b4e8_4998_9a50_ec16a58ef060.slice/crio-b4057aa9df5286c2f9e40560b9f1c51eeac18dbfcaf91de822891f51882148e6 WatchSource:0}: Error finding container b4057aa9df5286c2f9e40560b9f1c51eeac18dbfcaf91de822891f51882148e6: Status 404 returned error can't find the container with id b4057aa9df5286c2f9e40560b9f1c51eeac18dbfcaf91de822891f51882148e6
Oct 30 22:11:29.946411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:29.946386    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:11:30.128115 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:11:30.128081    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19e56bd0_5b23_4b22_98da_d4bd261aeea6.slice/crio-1f401e4b9a64df5cf7f71e1fb8edbe9f079b9033510743aa8e3976c81c1a8578 WatchSource:0}: Error finding container 1f401e4b9a64df5cf7f71e1fb8edbe9f079b9033510743aa8e3976c81c1a8578: Status 404 returned error can't find the container with id 1f401e4b9a64df5cf7f71e1fb8edbe9f079b9033510743aa8e3976c81c1a8578
Oct 30 22:11:30.130205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.130171    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:11:30.735670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.735628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" event={"ID":"19e56bd0-5b23-4b22-98da-d4bd261aeea6","Type":"ContainerStarted","Data":"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"}
Oct 30 22:11:30.735887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.735690    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" event={"ID":"19e56bd0-5b23-4b22-98da-d4bd261aeea6","Type":"ContainerStarted","Data":"1f401e4b9a64df5cf7f71e1fb8edbe9f079b9033510743aa8e3976c81c1a8578"}
Oct 30 22:11:30.735887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.735748    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:30.737425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.737402    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" event={"ID":"a0057df5-b4e8-4998-9a50-ec16a58ef060","Type":"ContainerStarted","Data":"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"}
Oct 30 22:11:30.737546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.737436    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" event={"ID":"a0057df5-b4e8-4998-9a50-ec16a58ef060","Type":"ContainerStarted","Data":"b4057aa9df5286c2f9e40560b9f1c51eeac18dbfcaf91de822891f51882148e6"}
Oct 30 22:11:30.737546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.737543    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:30.854529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.854471    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" podStartSLOduration=1.854453371 podStartE2EDuration="1.854453371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:30.851886719 +0000 UTC m=+2014.025211145" watchObservedRunningTime="2026-10-30 22:11:30.854453371 +0000 UTC m=+2014.027777798"
Oct 30 22:11:30.970723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:30.970654    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" podStartSLOduration=1.970636538 podStartE2EDuration="1.970636538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:30.968531597 +0000 UTC m=+2014.141856024" watchObservedRunningTime="2026-10-30 22:11:30.970636538 +0000 UTC m=+2014.143960993"
Oct 30 22:11:38.894330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:38.894287    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:11:38.899044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:38.899017    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:38.955254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:38.955212    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:11:38.999022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:38.998972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.000797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.000767    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j4x95\" (UniqueName: \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.001185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.001159    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95b84c\""
Oct 30 22:11:39.001297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.001281    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95b84c-dockercfg-56jp5\""
Oct 30 22:11:39.022745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.022709    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:11:39.028760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.028736    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.084101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.084069    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:11:39.102077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.101869    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrq67\" (UniqueName: \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.102077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.102014    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j4x95\" (UniqueName: \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.102379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.102171    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.102379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.102252    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.103257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.103226    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.164530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.164457    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j4x95\" (UniqueName: \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.202944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.202905    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.203158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.202960    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrq67\" (UniqueName: \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.203620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.203580    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.215204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.215176    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.265236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.265203    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrq67\" (UniqueName: \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\") pod \"payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.343507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.343470    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:39.589897 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:11:39.589860    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d1d3a14_0a2a_4c1b_88cc_f2e3b96bd501.slice/crio-5e04436fe9f1606a2b1cebcd20ea238b04b5e5c7949522793ebedf63e061f13c WatchSource:0}: Error finding container 5e04436fe9f1606a2b1cebcd20ea238b04b5e5c7949522793ebedf63e061f13c: Status 404 returned error can't find the container with id 5e04436fe9f1606a2b1cebcd20ea238b04b5e5c7949522793ebedf63e061f13c
Oct 30 22:11:39.592720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.592686    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:11:39.719360 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:11:39.719330    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1fd6397d_aed2_4a60_92fd_010813fa86ba.slice/crio-a6274df8ceea3daed1753b06e18e25b5b343ddb965a88e639a08624bde3788d0 WatchSource:0}: Error finding container a6274df8ceea3daed1753b06e18e25b5b343ddb965a88e639a08624bde3788d0: Status 404 returned error can't find the container with id a6274df8ceea3daed1753b06e18e25b5b343ddb965a88e639a08624bde3788d0
Oct 30 22:11:39.721318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.721294    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:11:39.800986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.800952    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" event={"ID":"1fd6397d-aed2-4a60-92fd-010813fa86ba","Type":"ContainerStarted","Data":"a6274df8ceea3daed1753b06e18e25b5b343ddb965a88e639a08624bde3788d0"}
Oct 30 22:11:39.802924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.802891    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" event={"ID":"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501","Type":"ContainerStarted","Data":"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"}
Oct 30 22:11:39.803056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.802949    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" event={"ID":"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501","Type":"ContainerStarted","Data":"5e04436fe9f1606a2b1cebcd20ea238b04b5e5c7949522793ebedf63e061f13c"}
Oct 30 22:11:39.803056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.802987    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:39.922001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:39.921883    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" podStartSLOduration=1.921863576 podStartE2EDuration="1.921863576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:39.921512501 +0000 UTC m=+2023.094836928" watchObservedRunningTime="2026-10-30 22:11:39.921863576 +0000 UTC m=+2023.095188003"
Oct 30 22:11:40.812042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:40.812006    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" event={"ID":"1fd6397d-aed2-4a60-92fd-010813fa86ba","Type":"ContainerStarted","Data":"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"}
Oct 30 22:11:40.812258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:40.812213    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:40.932464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:40.932414    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" podStartSLOduration=2.932400136 podStartE2EDuration="2.932400136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:11:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:11:40.929713594 +0000 UTC m=+2024.103038024" watchObservedRunningTime="2026-10-30 22:11:40.932400136 +0000 UTC m=+2024.105724562"
Oct 30 22:11:41.747974 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:41.747941    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:11:41.748171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:41.748026    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:11:50.811451 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:11:50.811402    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:11:50.815368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:50.815341    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:11:51.822177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:51.822148    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:11:52.955617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:52.955575    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:11:57.956026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:57.955956    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" podUID="97fabdc8-1e5f-498e-b341-327185f2a676" containerName="payload-processing" containerID="cri-o://32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be" gracePeriod=25
Oct 30 22:11:58.239911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.239885    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:11:58.245137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.245113    4882 scope.go:122] "RemoveContainer" containerID="744e3d0be9d94d9153ebbebd654eb833fb1f99aaa36cb5efaba0e96ed68b09ea"
Oct 30 22:11:58.247313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.247292    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\") pod \"97fabdc8-1e5f-498e-b341-327185f2a676\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") "
Oct 30 22:11:58.247475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.247461    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\" (UniqueName: \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\") pod \"97fabdc8-1e5f-498e-b341-327185f2a676\" (UID: \"97fabdc8-1e5f-498e-b341-327185f2a676\") "
Oct 30 22:11:58.247709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.247679    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume" pod "97fabdc8-1e5f-498e-b341-327185f2a676" (UID: "97fabdc8-1e5f-498e-b341-327185f2a676"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:11:58.247949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.247916    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97fabdc8-1e5f-498e-b341-327185f2a676-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:58.249540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.249519    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq" pod "97fabdc8-1e5f-498e-b341-327185f2a676" (UID: "97fabdc8-1e5f-498e-b341-327185f2a676"). InnerVolumeSpecName "kube-api-access-cfwnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:11:58.349368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.349319    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cfwnq\" (UniqueName: \"kubernetes.io/projected/97fabdc8-1e5f-498e-b341-327185f2a676-kube-api-access-cfwnq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:11:58.945281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.945247    4882 generic.go:356] "Generic (PLEG): container finished" podID="97fabdc8-1e5f-498e-b341-327185f2a676" containerID="32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be" exitCode=0
Oct 30 22:11:58.945465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.945350    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"
Oct 30 22:11:58.946783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.946747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" event={"ID":"97fabdc8-1e5f-498e-b341-327185f2a676","Type":"ContainerDied","Data":"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"}
Oct 30 22:11:58.946932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.946806    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz" event={"ID":"97fabdc8-1e5f-498e-b341-327185f2a676","Type":"ContainerDied","Data":"6f4a33d411b92121ecb6210a98585d606abba9dc6616d266d4f71df83aa121b0"}
Oct 30 22:11:58.946932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.946839    4882 scope.go:122] "RemoveContainer" containerID="32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"
Oct 30 22:11:58.957196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.957050    4882 scope.go:122] "RemoveContainer" containerID="32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"
Oct 30 22:11:58.957441 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:11:58.957369    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be\": container with ID starting with 32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be not found: ID does not exist" containerID="32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"
Oct 30 22:11:58.957441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:58.957396    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be"} err="failed to get container status \"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be\": rpc error: code = NotFound desc = could not find container \"32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be\": container with ID starting with 32f67b6fe2171fcfed21b32854a812f4fd5cc95bc56bc6df7ba23e5eef8dd1be not found: ID does not exist"
Oct 30 22:11:59.116866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:59.116830    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:11:59.120851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:11:59.120825    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-787f8c5877-kdtgz"]
Oct 30 22:12:00.943721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:00.943688    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97fabdc8-1e5f-498e-b341-327185f2a676" path="/var/lib/kubelet/pods/97fabdc8-1e5f-498e-b341-327185f2a676/volumes"
Oct 30 22:12:12.554920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:12.554778    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:12:12.652534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:12.652488    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:12:12.828026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:12.827942    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:12:17.556101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.556054    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" podUID="6b841170-148b-4fc4-a0fa-999d491259c4" containerName="payload-processing" containerID="cri-o://7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6" gracePeriod=25
Oct 30 22:12:17.653708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.653664    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" podUID="52ab1f9d-bb5d-46de-a623-a79b121506d6" containerName="payload-pre-processing" containerID="cri-o://afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d" gracePeriod=25
Oct 30 22:12:17.841383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.841348    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:12:17.948619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.948578    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:12:17.981666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.981629    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\" (UniqueName: \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\") pod \"6b841170-148b-4fc4-a0fa-999d491259c4\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") "
Oct 30 22:12:17.981873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.981713    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\") pod \"6b841170-148b-4fc4-a0fa-999d491259c4\" (UID: \"6b841170-148b-4fc4-a0fa-999d491259c4\") "
Oct 30 22:12:17.982150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.982124    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume" pod "6b841170-148b-4fc4-a0fa-999d491259c4" (UID: "6b841170-148b-4fc4-a0fa-999d491259c4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:17.982315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.982297    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6b841170-148b-4fc4-a0fa-999d491259c4-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:17.983978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:17.983954    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c" pod "6b841170-148b-4fc4-a0fa-999d491259c4" (UID: "6b841170-148b-4fc4-a0fa-999d491259c4"). InnerVolumeSpecName "kube-api-access-fq22c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:18.082851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.082759    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\") pod \"52ab1f9d-bb5d-46de-a623-a79b121506d6\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") "
Oct 30 22:12:18.083072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.082970    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\" (UniqueName: \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\") pod \"52ab1f9d-bb5d-46de-a623-a79b121506d6\" (UID: \"52ab1f9d-bb5d-46de-a623-a79b121506d6\") "
Oct 30 22:12:18.083505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083482    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq22c\" (UniqueName: \"kubernetes.io/projected/6b841170-148b-4fc4-a0fa-999d491259c4-kube-api-access-fq22c\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:18.083678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083578    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume" pod "52ab1f9d-bb5d-46de-a623-a79b121506d6" (UID: "52ab1f9d-bb5d-46de-a623-a79b121506d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:18.083678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083649    4882 generic.go:356] "Generic (PLEG): container finished" podID="52ab1f9d-bb5d-46de-a623-a79b121506d6" containerID="afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d" exitCode=0
Oct 30 22:12:18.083832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083708    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"
Oct 30 22:12:18.083832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083768    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" event={"ID":"52ab1f9d-bb5d-46de-a623-a79b121506d6","Type":"ContainerDied","Data":"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"}
Oct 30 22:12:18.083832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083810    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7" event={"ID":"52ab1f9d-bb5d-46de-a623-a79b121506d6","Type":"ContainerDied","Data":"897ec8d9ed37961e3d134f28a60888ad9779a803d3bd7571397d7975f92ce5b8"}
Oct 30 22:12:18.084011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.083840    4882 scope.go:122] "RemoveContainer" containerID="afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"
Oct 30 22:12:18.085940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.085911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs" pod "52ab1f9d-bb5d-46de-a623-a79b121506d6" (UID: "52ab1f9d-bb5d-46de-a623-a79b121506d6"). InnerVolumeSpecName "kube-api-access-qnmjs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:18.086066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.086021    4882 generic.go:356] "Generic (PLEG): container finished" podID="6b841170-148b-4fc4-a0fa-999d491259c4" containerID="7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6" exitCode=0
Oct 30 22:12:18.086134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.086070    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"
Oct 30 22:12:18.086134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.086108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" event={"ID":"6b841170-148b-4fc4-a0fa-999d491259c4","Type":"ContainerDied","Data":"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"}
Oct 30 22:12:18.086252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.086169    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv" event={"ID":"6b841170-148b-4fc4-a0fa-999d491259c4","Type":"ContainerDied","Data":"98cc3eb2733d92422f4cff77f37b5f89f191606f1b5027557cf7dc61eb4a6f0b"}
Oct 30 22:12:18.097337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.097312    4882 scope.go:122] "RemoveContainer" containerID="afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"
Oct 30 22:12:18.097785 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:18.097762    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d\": container with ID starting with afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d not found: ID does not exist" containerID="afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"
Oct 30 22:12:18.097909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.097792    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d"} err="failed to get container status \"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d\": rpc error: code = NotFound desc = could not find container \"afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d\": container with ID starting with afd620f7f2e06c6ca4a55bcfec840cf62b15a8e0bcb64db3ffdeb7d829bc767d not found: ID does not exist"
Oct 30 22:12:18.097909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.097811    4882 scope.go:122] "RemoveContainer" containerID="7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"
Oct 30 22:12:18.109717 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.109693    4882 scope.go:122] "RemoveContainer" containerID="7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"
Oct 30 22:12:18.110188 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:18.110158    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6\": container with ID starting with 7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6 not found: ID does not exist" containerID="7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"
Oct 30 22:12:18.110295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.110195    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6"} err="failed to get container status \"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6\": rpc error: code = NotFound desc = could not find container \"7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6\": container with ID starting with 7441bc4264af3fda5844476eded947bea562838a7a7c4d34023c1ecabc181ce6 not found: ID does not exist"
Oct 30 22:12:18.184644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.184568    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qnmjs\" (UniqueName: \"kubernetes.io/projected/52ab1f9d-bb5d-46de-a623-a79b121506d6-kube-api-access-qnmjs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:18.184644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.184641    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52ab1f9d-bb5d-46de-a623-a79b121506d6-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:18.252388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.252349    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:12:18.256451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.256427    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-6475cc7df9-pw9qv"]
Oct 30 22:12:18.506549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.506514    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:12:18.511767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.511736    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-79df5b7dcf-n6tw7"]
Oct 30 22:12:18.944091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.944053    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52ab1f9d-bb5d-46de-a623-a79b121506d6" path="/var/lib/kubelet/pods/52ab1f9d-bb5d-46de-a623-a79b121506d6/volumes"
Oct 30 22:12:18.944702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:18.944672    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b841170-148b-4fc4-a0fa-999d491259c4" path="/var/lib/kubelet/pods/6b841170-148b-4fc4-a0fa-999d491259c4/volumes"
Oct 30 22:12:20.808233 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:20.808181    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:12:33.982721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.982679    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"]
Oct 30 22:12:33.983493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983471    4882 cpu_manager.go:414] "containerMap: removing container" podUID="52ab1f9d-bb5d-46de-a623-a79b121506d6" containerName="payload-pre-processing"
Oct 30 22:12:33.983611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983495    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52ab1f9d-bb5d-46de-a623-a79b121506d6" containerName="payload-pre-processing"
Oct 30 22:12:33.983611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983540    4882 cpu_manager.go:414] "containerMap: removing container" podUID="97fabdc8-1e5f-498e-b341-327185f2a676" containerName="payload-processing"
Oct 30 22:12:33.983611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983550    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97fabdc8-1e5f-498e-b341-327185f2a676" containerName="payload-processing"
Oct 30 22:12:33.983611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983572    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6b841170-148b-4fc4-a0fa-999d491259c4" containerName="payload-processing"
Oct 30 22:12:33.983611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983580    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b841170-148b-4fc4-a0fa-999d491259c4" containerName="payload-processing"
Oct 30 22:12:33.983875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983741    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b841170-148b-4fc4-a0fa-999d491259c4" containerName="payload-processing"
Oct 30 22:12:33.983875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983765    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="52ab1f9d-bb5d-46de-a623-a79b121506d6" containerName="payload-pre-processing"
Oct 30 22:12:33.983875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.983782    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="97fabdc8-1e5f-498e-b341-327185f2a676" containerName="payload-processing"
Oct 30 22:12:33.987779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:33.987757    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.042198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.042164    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"]
Oct 30 22:12:34.069023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.068972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.069226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.069083    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.069226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.069185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7gbj\" (UniqueName: \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.088759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.088723    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:12:34.088759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.088739    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:12:34.170769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.170736    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.170942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.170842    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.170942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.170866    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7gbj\" (UniqueName: \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.171513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.171482    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.173101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.173077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.233047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.232975    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7gbj\" (UniqueName: \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\") pod \"maas-api-revoke-keys-e2e-api-a-52f191-f9xwn\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.305235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.305196    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:34.682516 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:12:34.682469    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4aff6c04_5e56_4a82_a412_993b7551f311.slice/crio-81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35 WatchSource:0}: Error finding container 81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35: Status 404 returned error can't find the container with id 81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35
Oct 30 22:12:34.683921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:34.683899    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"]
Oct 30 22:12:35.198259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:35.198167    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn" event={"ID":"4aff6c04-5e56-4a82-a412-993b7551f311","Type":"ContainerStarted","Data":"2f517e8715d00e2a02bbc9d40d3d9b66f4fe67bc2cd2937c6085c2d908f72f1b"}
Oct 30 22:12:35.198259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:35.198225    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn" event={"ID":"4aff6c04-5e56-4a82-a412-993b7551f311","Type":"ContainerStarted","Data":"81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35"}
Oct 30 22:12:35.317691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:35.317634    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn" podStartSLOduration=2.317619912 podStartE2EDuration="2.317619912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:35.315256295 +0000 UTC m=+2078.488580722" watchObservedRunningTime="2026-10-30 22:12:35.317619912 +0000 UTC m=+2078.490944338"
Oct 30 22:12:36.205550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:36.205514    4882 generic.go:356] "Generic (PLEG): container finished" podID="4aff6c04-5e56-4a82-a412-993b7551f311" containerID="2f517e8715d00e2a02bbc9d40d3d9b66f4fe67bc2cd2937c6085c2d908f72f1b" exitCode=0
Oct 30 22:12:36.205973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:36.205572    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn" event={"ID":"4aff6c04-5e56-4a82-a412-993b7551f311","Type":"ContainerDied","Data":"2f517e8715d00e2a02bbc9d40d3d9b66f4fe67bc2cd2937c6085c2d908f72f1b"}
Oct 30 22:12:37.359721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.359694    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:37.416064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.416024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\") pod \"4aff6c04-5e56-4a82-a412-993b7551f311\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") "
Oct 30 22:12:37.416247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.416233    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\") pod \"4aff6c04-5e56-4a82-a412-993b7551f311\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") "
Oct 30 22:12:37.416290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.416278    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\" (UniqueName: \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\") pod \"4aff6c04-5e56-4a82-a412-993b7551f311\" (UID: \"4aff6c04-5e56-4a82-a412-993b7551f311\") "
Oct 30 22:12:37.416420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.416395    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca" pod "4aff6c04-5e56-4a82-a412-993b7551f311" (UID: "4aff6c04-5e56-4a82-a412-993b7551f311"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:37.416784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.416764    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:37.418311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.418287    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj" pod "4aff6c04-5e56-4a82-a412-993b7551f311" (UID: "4aff6c04-5e56-4a82-a412-993b7551f311"). InnerVolumeSpecName "kube-api-access-t7gbj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:37.418417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.418357    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca" pod "4aff6c04-5e56-4a82-a412-993b7551f311" (UID: "4aff6c04-5e56-4a82-a412-993b7551f311"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:12:37.525192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.525094    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4aff6c04-5e56-4a82-a412-993b7551f311-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:37.525192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:37.525136    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7gbj\" (UniqueName: \"kubernetes.io/projected/4aff6c04-5e56-4a82-a412-993b7551f311-kube-api-access-t7gbj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:38.219998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:38.219966    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"
Oct 30 22:12:38.220171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:38.220016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn" event={"ID":"4aff6c04-5e56-4a82-a412-993b7551f311","Type":"ContainerDied","Data":"81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35"}
Oct 30 22:12:38.220171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:38.220066    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81be466abafd1c0a6c01a44509643674f075a12fb4640111a5270d1a755ece35"
Oct 30 22:12:45.492004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:45.491971    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:12:45.604756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:45.604722    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:12:46.686885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.686845    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:12:46.687431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.687417    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4aff6c04-5e56-4a82-a412-993b7551f311" containerName="revoke-keys"
Oct 30 22:12:46.687474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.687432    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4aff6c04-5e56-4a82-a412-993b7551f311" containerName="revoke-keys"
Oct 30 22:12:46.687563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.687554    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4aff6c04-5e56-4a82-a412-993b7551f311" containerName="revoke-keys"
Oct 30 22:12:46.691454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.691436    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:46.746271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.746222    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-f0b8c9\""
Oct 30 22:12:46.747326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.747296    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:12:46.792899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.792865    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-f0b8c9-dockercfg-p4hqt\""
Oct 30 22:12:46.841480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.841440    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h272\" (UniqueName: \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:46.841707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.841503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:46.942498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.942417    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h272\" (UniqueName: \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:46.942498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.942481    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:46.943431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:46.943405    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:47.005970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.005936    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h272\" (UniqueName: \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\") pod \"payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:47.305609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.305495    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:47.505141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.505102    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:12:47.510527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.510496    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.548538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.548500    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.548749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.548570    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q9mv9\" (UniqueName: \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.566768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.566685    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:12:47.649321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.649287    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.649504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.649351    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q9mv9\" (UniqueName: \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.650170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.650149    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.720081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.720037    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q9mv9\" (UniqueName: \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\") pod \"payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:47.779700 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:12:47.779658    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedcb6f31_fe79_4529_8ccb_3141caeaf8c3.slice/crio-11acf7ddd348cce57710078b3d5248e0b10fee5ab4f0bc0861a6c2d8a62a1a28 WatchSource:0}: Error finding container 11acf7ddd348cce57710078b3d5248e0b10fee5ab4f0bc0861a6c2d8a62a1a28: Status 404 returned error can't find the container with id 11acf7ddd348cce57710078b3d5248e0b10fee5ab4f0bc0861a6c2d8a62a1a28
Oct 30 22:12:47.781847 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.781828    4882 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Oct 30 22:12:47.782246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.782223    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:12:47.827445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:47.827348    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:48.202264 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:12:48.202219    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7663bf3_860d_4b41_9bed_3794c2399651.slice/crio-c09738cb5fabc92a789b6623086e4ef1c89ed54ee1cfba6a6478c8fa3569b5d4 WatchSource:0}: Error finding container c09738cb5fabc92a789b6623086e4ef1c89ed54ee1cfba6a6478c8fa3569b5d4: Status 404 returned error can't find the container with id c09738cb5fabc92a789b6623086e4ef1c89ed54ee1cfba6a6478c8fa3569b5d4
Oct 30 22:12:48.204848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:48.204825    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:12:48.284599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:48.284559    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" event={"ID":"e7663bf3-860d-4b41-9bed-3794c2399651","Type":"ContainerStarted","Data":"c09738cb5fabc92a789b6623086e4ef1c89ed54ee1cfba6a6478c8fa3569b5d4"}
Oct 30 22:12:48.285927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:48.285891    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" event={"ID":"edcb6f31-fe79-4529-8ccb-3141caeaf8c3","Type":"ContainerStarted","Data":"11acf7ddd348cce57710078b3d5248e0b10fee5ab4f0bc0861a6c2d8a62a1a28"}
Oct 30 22:12:50.305817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.305769    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" event={"ID":"e7663bf3-860d-4b41-9bed-3794c2399651","Type":"ContainerStarted","Data":"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"}
Oct 30 22:12:50.306289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.305881    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:12:50.307491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.307462    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" event={"ID":"edcb6f31-fe79-4529-8ccb-3141caeaf8c3","Type":"ContainerStarted","Data":"736f4b85d483882b109e7bee5a5ce5f5c2a683af80dbb51f49acb8aa7b1ebe0e"}
Oct 30 22:12:50.307636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.307530    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:12:50.424184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.424130    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" podStartSLOduration=1.879178321 podStartE2EDuration="3.424115674s" totalImagesPullingTime="1.544937353s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:50.423411415 +0000 UTC m=+2093.596735842" watchObservedRunningTime="2026-10-30 22:12:50.424115674 +0000 UTC m=+2093.597440101"
Oct 30 22:12:50.492656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.492614    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" podUID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" containerName="payload-processing" containerID="cri-o://7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358" gracePeriod=25
Oct 30 22:12:50.548054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.543791    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" podStartSLOduration=2.715470459 podStartE2EDuration="4.543767296s" totalImagesPullingTime="1.828296837s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:12:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:12:50.540282074 +0000 UTC m=+2093.713606517" watchObservedRunningTime="2026-10-30 22:12:50.543767296 +0000 UTC m=+2093.717091724"
Oct 30 22:12:50.605250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.605205    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" podUID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" containerName="payload-pre-processing" containerID="cri-o://94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0" gracePeriod=25
Oct 30 22:12:50.799631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.798411    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:12:50.817141 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:50.817103    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:12:50.884245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.884214    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\") pod \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") "
Oct 30 22:12:50.884425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.884413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\" (UniqueName: \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\") pod \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\" (UID: \"d9b26dab-b667-4cb0-8d8b-343e2a098d06\") "
Oct 30 22:12:50.884576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.884553    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume" pod "d9b26dab-b667-4cb0-8d8b-343e2a098d06" (UID: "d9b26dab-b667-4cb0-8d8b-343e2a098d06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:50.884732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.884716    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9b26dab-b667-4cb0-8d8b-343e2a098d06-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:50.886529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.886494    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92" pod "d9b26dab-b667-4cb0-8d8b-343e2a098d06" (UID: "d9b26dab-b667-4cb0-8d8b-343e2a098d06"). InnerVolumeSpecName "kube-api-access-sdm92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:50.915853 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.915819    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:12:50.985884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.985840    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\") pod \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") "
Oct 30 22:12:50.986064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.985935    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\" (UniqueName: \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\") pod \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\" (UID: \"3907bad2-d4c1-4f0a-b98b-343e32c050f3\") "
Oct 30 22:12:50.986279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.986250    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume" pod "3907bad2-d4c1-4f0a-b98b-343e32c050f3" (UID: "3907bad2-d4c1-4f0a-b98b-343e32c050f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:12:50.986279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.986264    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sdm92\" (UniqueName: \"kubernetes.io/projected/d9b26dab-b667-4cb0-8d8b-343e2a098d06-kube-api-access-sdm92\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:50.988041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:50.988019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc" pod "3907bad2-d4c1-4f0a-b98b-343e32c050f3" (UID: "3907bad2-d4c1-4f0a-b98b-343e32c050f3"). InnerVolumeSpecName "kube-api-access-q4jwc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:12:51.087941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.087906    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4jwc\" (UniqueName: \"kubernetes.io/projected/3907bad2-d4c1-4f0a-b98b-343e32c050f3-kube-api-access-q4jwc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:51.087941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.087937    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3907bad2-d4c1-4f0a-b98b-343e32c050f3-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:12:51.315630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.315525    4882 generic.go:356] "Generic (PLEG): container finished" podID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" containerID="7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358" exitCode=0
Oct 30 22:12:51.315630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.315598    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"
Oct 30 22:12:51.316164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.315628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" event={"ID":"d9b26dab-b667-4cb0-8d8b-343e2a098d06","Type":"ContainerDied","Data":"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"}
Oct 30 22:12:51.316164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.315692    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk" event={"ID":"d9b26dab-b667-4cb0-8d8b-343e2a098d06","Type":"ContainerDied","Data":"8382451abd7feca6f0e4a8444a4a24ce7d8a2ee232b0877c310131dcc6057ce0"}
Oct 30 22:12:51.316164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.315718    4882 scope.go:122] "RemoveContainer" containerID="7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"
Oct 30 22:12:51.317039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.317012    4882 generic.go:356] "Generic (PLEG): container finished" podID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" containerID="94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0" exitCode=0
Oct 30 22:12:51.317158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.317063    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"
Oct 30 22:12:51.317158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.317106    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" event={"ID":"3907bad2-d4c1-4f0a-b98b-343e32c050f3","Type":"ContainerDied","Data":"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"}
Oct 30 22:12:51.317272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.317160    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm" event={"ID":"3907bad2-d4c1-4f0a-b98b-343e32c050f3","Type":"ContainerDied","Data":"e29cf8c18927afc78782a238136556a39ded13b354d3902fc945f1ffbf1bc292"}
Oct 30 22:12:51.326453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.326432    4882 scope.go:122] "RemoveContainer" containerID="7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"
Oct 30 22:12:51.326779 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:51.326757    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358\": container with ID starting with 7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358 not found: ID does not exist" containerID="7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"
Oct 30 22:12:51.326851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.326792    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358"} err="failed to get container status \"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358\": rpc error: code = NotFound desc = could not find container \"7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358\": container with ID starting with 7c871b7dee6ed3f09005a81ccc9ed1700115c1946c9642473c4be011eb124358 not found: ID does not exist"
Oct 30 22:12:51.326851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.326811    4882 scope.go:122] "RemoveContainer" containerID="94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"
Oct 30 22:12:51.336234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.336214    4882 scope.go:122] "RemoveContainer" containerID="94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"
Oct 30 22:12:51.336549 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:12:51.336525    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0\": container with ID starting with 94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0 not found: ID does not exist" containerID="94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"
Oct 30 22:12:51.336646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.336555    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0"} err="failed to get container status \"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0\": rpc error: code = NotFound desc = could not find container \"94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0\": container with ID starting with 94dccaf4f6cb99236139fe267a7b4417c9103a409044bbd10fc6cb6ba18127f0 not found: ID does not exist"
Oct 30 22:12:51.483888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.483851    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:12:51.488031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.488001    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-52f191-fdd756487-t6lgk"]
Oct 30 22:12:51.600901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.600871    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:12:51.606621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:51.606580    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-52f191-6ccb58fc47-sdzsm"]
Oct 30 22:12:52.945069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:52.945037    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" path="/var/lib/kubelet/pods/3907bad2-d4c1-4f0a-b98b-343e32c050f3/volumes"
Oct 30 22:12:52.945502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:52.945485    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" path="/var/lib/kubelet/pods/d9b26dab-b667-4cb0-8d8b-343e2a098d06/volumes"
Oct 30 22:12:57.222730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:57.222694    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:12:57.232718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:57.232694    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:12:58.271443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:12:58.271410    4882 scope.go:122] "RemoveContainer" containerID="8b1faa2987d8ea1e347ef8d8b712a3f93460b6bb338837e5ee223975582be5cb"
Oct 30 22:13:00.009455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:00.009416    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:13:01.319991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:01.319962    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:13:01.320392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:01.320070    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:13:02.836284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836249    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:13:02.836845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836830    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" containerName="payload-processing"
Oct 30 22:13:02.836898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836846    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" containerName="payload-processing"
Oct 30 22:13:02.836898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836862    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" containerName="payload-pre-processing"
Oct 30 22:13:02.836898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836868    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" containerName="payload-pre-processing"
Oct 30 22:13:02.836999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836951    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9b26dab-b667-4cb0-8d8b-343e2a098d06" containerName="payload-processing"
Oct 30 22:13:02.836999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.836959    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3907bad2-d4c1-4f0a-b98b-343e32c050f3" containerName="payload-pre-processing"
Oct 30 22:13:02.842185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.842161    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:02.900488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.900438    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-65659d\""
Oct 30 22:13:02.900803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.900771    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-65659d-dockercfg-vqsd6\""
Oct 30 22:13:02.926127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.926091    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:02.926337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.926135    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9l77\" (UniqueName: \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:02.954444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:02.954415    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:13:03.027467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.027426    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:03.027710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.027508    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9l77\" (UniqueName: \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:03.028251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.028231    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:03.102200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.102120    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9l77\" (UniqueName: \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\") pod \"payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:03.157377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.157343    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:03.551221 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:03.551175    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ca187ec_f7a7_46fe_ad63_7812ddc13f2c.slice/crio-24032ef9169ca4d34187b395ccf19f91d491d820c8d284d35432f0bb8ff9bd83 WatchSource:0}: Error finding container 24032ef9169ca4d34187b395ccf19f91d491d820c8d284d35432f0bb8ff9bd83: Status 404 returned error can't find the container with id 24032ef9169ca4d34187b395ccf19f91d491d820c8d284d35432f0bb8ff9bd83
Oct 30 22:13:03.551354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.551235    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:13:03.611862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.611830    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:13:03.616183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.616162    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.671430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.671401    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:13:03.735583 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.735518    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.735839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.735704    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f76d8\" (UniqueName: \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.837169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.837133    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.837674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.837209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f76d8\" (UniqueName: \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.837976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.837955    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.899391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.899349    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f76d8\" (UniqueName: \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\") pod \"payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:03.930413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:03.930381    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:04.307086 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:04.307051    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda26faa2b_bfec_4c15_9f2f_e029abf94670.slice/crio-34a45ee0c491e8052098bb7be50233853d8057811bcdde7c66c7c10ba90789ac WatchSource:0}: Error finding container 34a45ee0c491e8052098bb7be50233853d8057811bcdde7c66c7c10ba90789ac: Status 404 returned error can't find the container with id 34a45ee0c491e8052098bb7be50233853d8057811bcdde7c66c7c10ba90789ac
Oct 30 22:13:04.309748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.309715    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:13:04.420296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.420249    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" event={"ID":"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c","Type":"ContainerStarted","Data":"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"}
Oct 30 22:13:04.420470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.420312    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" event={"ID":"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c","Type":"ContainerStarted","Data":"24032ef9169ca4d34187b395ccf19f91d491d820c8d284d35432f0bb8ff9bd83"}
Oct 30 22:13:04.420470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.420344    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:04.422154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.422122    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerStarted","Data":"8fc7c41fc3de9c688545f1fe480b76a8a0facfc11d122ff574367aceeb6b1651"}
Oct 30 22:13:04.422293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.422178    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerStarted","Data":"34a45ee0c491e8052098bb7be50233853d8057811bcdde7c66c7c10ba90789ac"}
Oct 30 22:13:04.422293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.422208    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:04.540907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.540839    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" podStartSLOduration=2.540818514 podStartE2EDuration="2.540818514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:04.537423271 +0000 UTC m=+2107.710747697" watchObservedRunningTime="2026-10-30 22:13:04.540818514 +0000 UTC m=+2107.714142946"
Oct 30 22:13:04.658175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:04.657879    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" podStartSLOduration=1.657858563 podStartE2EDuration="1.657858563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:04.65739124 +0000 UTC m=+2107.830715666" watchObservedRunningTime="2026-10-30 22:13:04.657858563 +0000 UTC m=+2107.831182971"
Oct 30 22:13:14.484557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:14.484520    4882 generic.go:356] "Generic (PLEG): container finished" podID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerID="8fc7c41fc3de9c688545f1fe480b76a8a0facfc11d122ff574367aceeb6b1651" exitCode=1
Oct 30 22:13:14.484992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:14.484572    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerDied","Data":"8fc7c41fc3de9c688545f1fe480b76a8a0facfc11d122ff574367aceeb6b1651"}
Oct 30 22:13:14.484992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:14.484973    4882 scope.go:122] "RemoveContainer" containerID="8fc7c41fc3de9c688545f1fe480b76a8a0facfc11d122ff574367aceeb6b1651"
Oct 30 22:13:15.040672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:15.040638    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 22:13:15.040900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:15.040850    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" containerID="cri-o://2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2" gracePeriod=30
Oct 30 22:13:15.431542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:15.431506    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:13:15.492500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:15.492463    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerStarted","Data":"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"}
Oct 30 22:13:15.492937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:15.492680    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:18.913111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:18.913072    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:13:18.913629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:18.913131    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:13:20.317789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.317710    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 22:13:20.338160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338116    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338190    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338260    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338307    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338378    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338415    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338443    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338506    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338544    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\") pod \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\" (UID: \"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a\") "
Oct 30 22:13:20.338737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.338698    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:20.339188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.339130    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:20.339188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.339177    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:20.339310 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.339195    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:20.339310 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.339225    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.339412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.339338    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:20.341275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.341231    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:20.341474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.341453    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:13:20.341745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.341722    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:13:20.341821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.341763    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff" pod "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" (UID: "ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a"). InnerVolumeSpecName "kube-api-access-pjcff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:20.440682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440644    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440672    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjcff\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-kube-api-access-pjcff\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440684    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440697    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440707    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440717    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440726    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.440927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.440734    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:20.526192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.526158    4882 generic.go:356] "Generic (PLEG): container finished" podID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerID="2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2" exitCode=0
Oct 30 22:13:20.526372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.526207    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"
Oct 30 22:13:20.526372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.526239    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" event={"ID":"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a","Type":"ContainerDied","Data":"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"}
Oct 30 22:13:20.526372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.526300    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf" event={"ID":"ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a","Type":"ContainerDied","Data":"cbbc982b5f971568904fb6d92dbe49ec3c3595257e0885edff9c2ce18b8cc60c"}
Oct 30 22:13:20.526372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.526328    4882 scope.go:122] "RemoveContainer" containerID="2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"
Oct 30 22:13:20.539111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.539075    4882 scope.go:122] "RemoveContainer" containerID="2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"
Oct 30 22:13:20.539468 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:20.539442    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2\": container with ID starting with 2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2 not found: ID does not exist" containerID="2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"
Oct 30 22:13:20.539565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.539480    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2"} err="failed to get container status \"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2\": rpc error: code = NotFound desc = could not find container \"2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2\": container with ID starting with 2fcb559298f0ebd622281ec94c0aec0032be5178b86837cbd365320c48e0afe2 not found: ID does not exist"
Oct 30 22:13:20.812118 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:20.812084    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:13:20.846035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.846002    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 22:13:20.896311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.896272    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-zjvbf"]
Oct 30 22:13:20.943334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:20.943297    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" path="/var/lib/kubelet/pods/ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a/volumes"
Oct 30 22:13:25.041005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.040914    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:13:25.148324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.148287    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:13:25.563519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.563482    4882 generic.go:356] "Generic (PLEG): container finished" podID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547" exitCode=1
Oct 30 22:13:25.563706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.563556    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerDied","Data":"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"}
Oct 30 22:13:25.563706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.563641    4882 scope.go:122] "RemoveContainer" containerID="8fc7c41fc3de9c688545f1fe480b76a8a0facfc11d122ff574367aceeb6b1651"
Oct 30 22:13:25.564093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:25.564074    4882 scope.go:122] "RemoveContainer" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"
Oct 30 22:13:25.564513 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:25.564490    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"payload-processing\" with CrashLoopBackOff: \"back-off 10s restarting failed container=payload-processing pod=payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7_openshift-ingress(a26faa2b-bfec-4c15-9f2f-e029abf94670)\"" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670"
Oct 30 22:13:29.571651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.571611    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:13:29.572206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.572192    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy"
Oct 30 22:13:29.572251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.572207    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy"
Oct 30 22:13:29.572334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.572325    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee5b1749-e7b8-4b38-bbb3-777b0a01bb1a" containerName="istio-proxy"
Oct 30 22:13:29.577150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.577132    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.630006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.629973    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:13:29.658630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.658514    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.659004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.658964    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24rcd\" (UniqueName: \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.687782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.687744    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c4bfac\""
Oct 30 22:13:29.688021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.688001    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c4bfac-dockercfg-r5ld5\""
Oct 30 22:13:29.744306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.744275    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:13:29.748361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.748338    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:29.760444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.760418    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.760572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.760453    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24rcd\" (UniqueName: \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.761120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.761101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.770570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.770539    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:13:29.823954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.823882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24rcd\" (UniqueName: \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.861429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.861382    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xgz9\" (UniqueName: \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:29.861634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.861466    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:29.890889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.890859    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:29.962515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.962473    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xgz9\" (UniqueName: \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:29.962697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.962628    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:29.964759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:29.963706    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:30.027305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.027267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xgz9\" (UniqueName: \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\") pod \"payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:30.042988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.042943    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" podUID="a0057df5-b4e8-4998-9a50-ec16a58ef060" containerName="payload-processing" containerID="cri-o://35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c" gracePeriod=25
Oct 30 22:13:30.062509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.062486    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:30.149353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.149303    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" podUID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" containerName="payload-pre-processing" containerID="cri-o://051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52" gracePeriod=25
Oct 30 22:13:30.277979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.277947    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:13:30.278518 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:30.278495    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a02045c_9b27_4263_a592_1dc5793f6123.slice/crio-f235912972218a0f1489f12935a47aa742281f1f2b409514437878e254d00dcd WatchSource:0}: Error finding container f235912972218a0f1489f12935a47aa742281f1f2b409514437878e254d00dcd: Status 404 returned error can't find the container with id f235912972218a0f1489f12935a47aa742281f1f2b409514437878e254d00dcd
Oct 30 22:13:30.304848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.304810    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:13:30.367189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.367143    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\" (UniqueName: \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\") pod \"a0057df5-b4e8-4998-9a50-ec16a58ef060\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") "
Oct 30 22:13:30.367345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.367254    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\") pod \"a0057df5-b4e8-4998-9a50-ec16a58ef060\" (UID: \"a0057df5-b4e8-4998-9a50-ec16a58ef060\") "
Oct 30 22:13:30.367702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.367670    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume" pod "a0057df5-b4e8-4998-9a50-ec16a58ef060" (UID: "a0057df5-b4e8-4998-9a50-ec16a58ef060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:30.367933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.367913    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0057df5-b4e8-4998-9a50-ec16a58ef060-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:30.369849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.369819    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985" pod "a0057df5-b4e8-4998-9a50-ec16a58ef060" (UID: "a0057df5-b4e8-4998-9a50-ec16a58ef060"). InnerVolumeSpecName "kube-api-access-4b985". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:30.415132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.415103    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:13:30.441306 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:30.441274    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ba52d04_6f37_4a45_90ca_0e85ea2a660c.slice/crio-5d41b1e5c85902e6a597b3f2d77d3e6c0e837f84a8cdcccefe9386adae5142a4 WatchSource:0}: Error finding container 5d41b1e5c85902e6a597b3f2d77d3e6c0e837f84a8cdcccefe9386adae5142a4: Status 404 returned error can't find the container with id 5d41b1e5c85902e6a597b3f2d77d3e6c0e837f84a8cdcccefe9386adae5142a4
Oct 30 22:13:30.468997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.468969    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4b985\" (UniqueName: \"kubernetes.io/projected/a0057df5-b4e8-4998-9a50-ec16a58ef060-kube-api-access-4b985\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:30.477256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.477226    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:13:30.570418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.570385    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\") pod \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") "
Oct 30 22:13:30.570642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.570475    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\" (UniqueName: \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\") pod \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\" (UID: \"19e56bd0-5b23-4b22-98da-d4bd261aeea6\") "
Oct 30 22:13:30.570879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.570853    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume" pod "19e56bd0-5b23-4b22-98da-d4bd261aeea6" (UID: "19e56bd0-5b23-4b22-98da-d4bd261aeea6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:30.572746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.572719    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f" pod "19e56bd0-5b23-4b22-98da-d4bd261aeea6" (UID: "19e56bd0-5b23-4b22-98da-d4bd261aeea6"). InnerVolumeSpecName "kube-api-access-2lv9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:30.599647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.599520    4882 generic.go:356] "Generic (PLEG): container finished" podID="a0057df5-b4e8-4998-9a50-ec16a58ef060" containerID="35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c" exitCode=0
Oct 30 22:13:30.599647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.599567    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"
Oct 30 22:13:30.599647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.599635    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" event={"ID":"a0057df5-b4e8-4998-9a50-ec16a58ef060","Type":"ContainerDied","Data":"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"}
Oct 30 22:13:30.599936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.599691    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr" event={"ID":"a0057df5-b4e8-4998-9a50-ec16a58ef060","Type":"ContainerDied","Data":"b4057aa9df5286c2f9e40560b9f1c51eeac18dbfcaf91de822891f51882148e6"}
Oct 30 22:13:30.599936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.599721    4882 scope.go:122] "RemoveContainer" containerID="35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"
Oct 30 22:13:30.601453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.601423    4882 generic.go:356] "Generic (PLEG): container finished" podID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" containerID="051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52" exitCode=0
Oct 30 22:13:30.601580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.601481    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"
Oct 30 22:13:30.601580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.601499    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" event={"ID":"19e56bd0-5b23-4b22-98da-d4bd261aeea6","Type":"ContainerDied","Data":"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"}
Oct 30 22:13:30.601580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.601550    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5" event={"ID":"19e56bd0-5b23-4b22-98da-d4bd261aeea6","Type":"ContainerDied","Data":"1f401e4b9a64df5cf7f71e1fb8edbe9f079b9033510743aa8e3976c81c1a8578"}
Oct 30 22:13:30.605901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.605871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" event={"ID":"3a02045c-9b27-4263-a592-1dc5793f6123","Type":"ContainerStarted","Data":"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"}
Oct 30 22:13:30.606027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.605925    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" event={"ID":"3a02045c-9b27-4263-a592-1dc5793f6123","Type":"ContainerStarted","Data":"f235912972218a0f1489f12935a47aa742281f1f2b409514437878e254d00dcd"}
Oct 30 22:13:30.606690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.606670    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:30.608818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.608608    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" event={"ID":"2ba52d04-6f37-4a45-90ca-0e85ea2a660c","Type":"ContainerStarted","Data":"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"}
Oct 30 22:13:30.608818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.608653    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" event={"ID":"2ba52d04-6f37-4a45-90ca-0e85ea2a660c","Type":"ContainerStarted","Data":"5d41b1e5c85902e6a597b3f2d77d3e6c0e837f84a8cdcccefe9386adae5142a4"}
Oct 30 22:13:30.608818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.608714    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:30.611061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.611036    4882 scope.go:122] "RemoveContainer" containerID="35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"
Oct 30 22:13:30.611330 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:30.611311    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c\": container with ID starting with 35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c not found: ID does not exist" containerID="35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"
Oct 30 22:13:30.611386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.611337    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c"} err="failed to get container status \"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c\": rpc error: code = NotFound desc = could not find container \"35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c\": container with ID starting with 35df2cc27e3030bbd23b35021ff80ebc1794509c50d1e5ca81338b67b694226c not found: ID does not exist"
Oct 30 22:13:30.611386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.611352    4882 scope.go:122] "RemoveContainer" containerID="051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"
Oct 30 22:13:30.620500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.620480    4882 scope.go:122] "RemoveContainer" containerID="051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"
Oct 30 22:13:30.620802 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:30.620784    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52\": container with ID starting with 051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52 not found: ID does not exist" containerID="051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"
Oct 30 22:13:30.620869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.620811    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52"} err="failed to get container status \"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52\": rpc error: code = NotFound desc = could not find container \"051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52\": container with ID starting with 051d95f0d34772ddda35a7f3b2e3c2a335214f926e0d12bc10d5bc0700965b52 not found: ID does not exist"
Oct 30 22:13:30.672049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.672018    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/19e56bd0-5b23-4b22-98da-d4bd261aeea6-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:30.672049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.672050    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lv9f\" (UniqueName: \"kubernetes.io/projected/19e56bd0-5b23-4b22-98da-d4bd261aeea6-kube-api-access-2lv9f\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:30.794199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.794163    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:13:30.851222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.851140    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-694cc756bc-jd5z5"]
Oct 30 22:13:30.924418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.924385    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:13:30.942998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.942964    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" path="/var/lib/kubelet/pods/19e56bd0-5b23-4b22-98da-d4bd261aeea6/volumes"
Oct 30 22:13:30.974234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:30.974198    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-a4257d-54486d8c5f-7fpkr"]
Oct 30 22:13:31.052557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:31.052505    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" podStartSLOduration=2.052488941 podStartE2EDuration="2.052488941s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:31.05122035 +0000 UTC m=+2134.224544779" watchObservedRunningTime="2026-10-30 22:13:31.052488941 +0000 UTC m=+2134.225813368"
Oct 30 22:13:31.222747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:31.222638    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" podStartSLOduration=2.222622141 podStartE2EDuration="2.222622141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:31.180005023 +0000 UTC m=+2134.353329450" watchObservedRunningTime="2026-10-30 22:13:31.222622141 +0000 UTC m=+2134.395946567"
Oct 30 22:13:32.944124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:32.944091    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0057df5-b4e8-4998-9a50-ec16a58ef060" path="/var/lib/kubelet/pods/a0057df5-b4e8-4998-9a50-ec16a58ef060/volumes"
Oct 30 22:13:37.000629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.000573    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:13:37.001529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001506    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a0057df5-b4e8-4998-9a50-ec16a58ef060" containerName="payload-processing"
Oct 30 22:13:37.001626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001530    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0057df5-b4e8-4998-9a50-ec16a58ef060" containerName="payload-processing"
Oct 30 22:13:37.001626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001577    4882 cpu_manager.go:414] "containerMap: removing container" podUID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" containerName="payload-pre-processing"
Oct 30 22:13:37.001626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001602    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" containerName="payload-pre-processing"
Oct 30 22:13:37.001794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001782    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0057df5-b4e8-4998-9a50-ec16a58ef060" containerName="payload-processing"
Oct 30 22:13:37.001829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.001803    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="19e56bd0-5b23-4b22-98da-d4bd261aeea6" containerName="payload-pre-processing"
Oct 30 22:13:37.007509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.007487    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.061037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.061002    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5d2b13\""
Oct 30 22:13:37.117060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.117024    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5d2b13-dockercfg-hwwqv\""
Oct 30 22:13:37.127894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.127284    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:13:37.127894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.127367    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:13:37.136101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.134699    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.149152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.149112    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.149318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.149197    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7q6\" (UniqueName: \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.181124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.180963    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:13:37.251036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.250948    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z95b8\" (UniqueName: \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.251036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.250989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.251232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.251202    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.251299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.251283    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7q6\" (UniqueName: \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.251985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.251967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.316153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.316123    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7q6\" (UniqueName: \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.324289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.324261    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:37.354325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.354285    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z95b8\" (UniqueName: \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.354524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.354342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.355497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.355289    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.418164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.418098    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z95b8\" (UniqueName: \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\") pod \"payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.459927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.459875    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:37.709329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.709294    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:13:37.735888 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:37.735848    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a42d1c_be1b_4a3f_aff3_5db39f0fbb1a.slice/crio-656817cdb24d52591f44ef8d53a259f9f3a1b1cf918c2a3b4a619497a7694650 WatchSource:0}: Error finding container 656817cdb24d52591f44ef8d53a259f9f3a1b1cf918c2a3b4a619497a7694650: Status 404 returned error can't find the container with id 656817cdb24d52591f44ef8d53a259f9f3a1b1cf918c2a3b4a619497a7694650
Oct 30 22:13:37.736842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.736809    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:13:37.827326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.827291    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:13:37.863431 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:37.863392    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0eced86d_2e75_48f9_a98f_218e397abc97.slice/crio-91e7b765e7227890e558ac06da549ebf587787ff01828bdbd5b9f6494a2db29b WatchSource:0}: Error finding container 91e7b765e7227890e558ac06da549ebf587787ff01828bdbd5b9f6494a2db29b: Status 404 returned error can't find the container with id 91e7b765e7227890e558ac06da549ebf587787ff01828bdbd5b9f6494a2db29b
Oct 30 22:13:37.880333 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:37.880309    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:13:38.667382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.667346    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" event={"ID":"0eced86d-2e75-48f9-a98f-218e397abc97","Type":"ContainerStarted","Data":"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"}
Oct 30 22:13:38.667895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.667401    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" event={"ID":"0eced86d-2e75-48f9-a98f-218e397abc97","Type":"ContainerStarted","Data":"91e7b765e7227890e558ac06da549ebf587787ff01828bdbd5b9f6494a2db29b"}
Oct 30 22:13:38.667895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.667529    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:38.669299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.669270    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" event={"ID":"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a","Type":"ContainerStarted","Data":"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"}
Oct 30 22:13:38.669425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.669310    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" event={"ID":"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a","Type":"ContainerStarted","Data":"656817cdb24d52591f44ef8d53a259f9f3a1b1cf918c2a3b4a619497a7694650"}
Oct 30 22:13:38.669425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.669343    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:38.788186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.788118    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" podStartSLOduration=1.788096052 podStartE2EDuration="1.788096052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:38.784306063 +0000 UTC m=+2141.957630490" watchObservedRunningTime="2026-10-30 22:13:38.788096052 +0000 UTC m=+2141.961420480"
Oct 30 22:13:38.901980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:38.901906    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" podStartSLOduration=2.901884833 podStartE2EDuration="2.901884833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:38.900648389 +0000 UTC m=+2142.073972817" watchObservedRunningTime="2026-10-30 22:13:38.901884833 +0000 UTC m=+2142.075209274"
Oct 30 22:13:39.936042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:39.936008    4882 scope.go:122] "RemoveContainer" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"
Oct 30 22:13:40.686191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:40.686148    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerStarted","Data":"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"}
Oct 30 22:13:40.686425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:40.686404    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:41.620399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:41.620370    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:13:42.627756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:42.627727    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:13:42.709910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:42.709837    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" podUID="1fd6397d-aed2-4a60-92fd-010813fa86ba" containerName="payload-processing" containerID="cri-o://1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522" gracePeriod=25
Oct 30 22:13:42.827996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:42.827956    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" podUID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" containerName="payload-pre-processing" containerID="cri-o://387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc" gracePeriod=25
Oct 30 22:13:42.987181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:42.987148    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:13:43.018626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.018569    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\" (UniqueName: \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\") pod \"1fd6397d-aed2-4a60-92fd-010813fa86ba\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") "
Oct 30 22:13:43.018794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.018659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\") pod \"1fd6397d-aed2-4a60-92fd-010813fa86ba\" (UID: \"1fd6397d-aed2-4a60-92fd-010813fa86ba\") "
Oct 30 22:13:43.019063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.019036    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume" pod "1fd6397d-aed2-4a60-92fd-010813fa86ba" (UID: "1fd6397d-aed2-4a60-92fd-010813fa86ba"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:43.019258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.019240    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1fd6397d-aed2-4a60-92fd-010813fa86ba-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:43.021407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.021373    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67" pod "1fd6397d-aed2-4a60-92fd-010813fa86ba" (UID: "1fd6397d-aed2-4a60-92fd-010813fa86ba"). InnerVolumeSpecName "kube-api-access-hrq67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:43.091924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.091895    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:13:43.120844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.120799    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\" (UniqueName: \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\") pod \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") "
Oct 30 22:13:43.121031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.120875    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\") pod \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\" (UID: \"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501\") "
Oct 30 22:13:43.121341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.121306    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume" pod "2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" (UID: "2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:13:43.121554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.121455    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrq67\" (UniqueName: \"kubernetes.io/projected/1fd6397d-aed2-4a60-92fd-010813fa86ba-kube-api-access-hrq67\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:43.121554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.121480    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:43.123145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.123123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95" pod "2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" (UID: "2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501"). InnerVolumeSpecName "kube-api-access-j4x95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:13:43.223377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.223289    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j4x95\" (UniqueName: \"kubernetes.io/projected/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501-kube-api-access-j4x95\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:13:43.714180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.714146    4882 generic.go:356] "Generic (PLEG): container finished" podID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" containerID="387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc" exitCode=0
Oct 30 22:13:43.714645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.714190    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"
Oct 30 22:13:43.714645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.714220    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" event={"ID":"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501","Type":"ContainerDied","Data":"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"}
Oct 30 22:13:43.714645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.714272    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv" event={"ID":"2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501","Type":"ContainerDied","Data":"5e04436fe9f1606a2b1cebcd20ea238b04b5e5c7949522793ebedf63e061f13c"}
Oct 30 22:13:43.714645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.714294    4882 scope.go:122] "RemoveContainer" containerID="387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"
Oct 30 22:13:43.715947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.715923    4882 generic.go:356] "Generic (PLEG): container finished" podID="1fd6397d-aed2-4a60-92fd-010813fa86ba" containerID="1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522" exitCode=0
Oct 30 22:13:43.716042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.715980    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"
Oct 30 22:13:43.716042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.716003    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" event={"ID":"1fd6397d-aed2-4a60-92fd-010813fa86ba","Type":"ContainerDied","Data":"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"}
Oct 30 22:13:43.716132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.716054    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z" event={"ID":"1fd6397d-aed2-4a60-92fd-010813fa86ba","Type":"ContainerDied","Data":"a6274df8ceea3daed1753b06e18e25b5b343ddb965a88e639a08624bde3788d0"}
Oct 30 22:13:43.736346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.736323    4882 scope.go:122] "RemoveContainer" containerID="387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"
Oct 30 22:13:43.736705 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:43.736680    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc\": container with ID starting with 387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc not found: ID does not exist" containerID="387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"
Oct 30 22:13:43.736772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.736714    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc"} err="failed to get container status \"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc\": rpc error: code = NotFound desc = could not find container \"387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc\": container with ID starting with 387a59b07babe8972dc4c6016d85cad906d0df4cbd86baa571bdd4b4479501cc not found: ID does not exist"
Oct 30 22:13:43.736772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.736732    4882 scope.go:122] "RemoveContainer" containerID="1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"
Oct 30 22:13:43.750420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.750398    4882 scope.go:122] "RemoveContainer" containerID="1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"
Oct 30 22:13:43.750773 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:43.750751    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522\": container with ID starting with 1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522 not found: ID does not exist" containerID="1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"
Oct 30 22:13:43.750849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.750781    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522"} err="failed to get container status \"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522\": rpc error: code = NotFound desc = could not find container \"1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522\": container with ID starting with 1f133af4592faed91ba109b5a688687439219a3b0395244e3a6e3079b2a9f522 not found: ID does not exist"
Oct 30 22:13:43.882002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.881936    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:13:43.931766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:43.931733    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-66d7f7f659-xggxv"]
Oct 30 22:13:44.048885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:44.048546    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:13:44.055401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:44.055370    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5fddcc88bc-7vp7z"]
Oct 30 22:13:44.943395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:44.943360    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fd6397d-aed2-4a60-92fd-010813fa86ba" path="/var/lib/kubelet/pods/1fd6397d-aed2-4a60-92fd-010813fa86ba/volumes"
Oct 30 22:13:44.943829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:44.943777    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" path="/var/lib/kubelet/pods/2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501/volumes"
Oct 30 22:13:45.743215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743182    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:13:45.743797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743779    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" containerName="payload-pre-processing"
Oct 30 22:13:45.743797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743797    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" containerName="payload-pre-processing"
Oct 30 22:13:45.743946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743836    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1fd6397d-aed2-4a60-92fd-010813fa86ba" containerName="payload-processing"
Oct 30 22:13:45.743946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743842    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fd6397d-aed2-4a60-92fd-010813fa86ba" containerName="payload-processing"
Oct 30 22:13:45.743946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743940    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fd6397d-aed2-4a60-92fd-010813fa86ba" containerName="payload-processing"
Oct 30 22:13:45.744062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.743954    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2d1d3a14-0a2a-4c1b-88cc-f2e3b96bd501" containerName="payload-pre-processing"
Oct 30 22:13:45.747426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.747347    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.805682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.805645    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:13:45.848766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.848728    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-138566\""
Oct 30 22:13:45.848941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.848821    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-138566-dockercfg-jbcjq\""
Oct 30 22:13:45.854298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.854271    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.854423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.854384    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6g9h\" (UniqueName: \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.936790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.936747    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:13:45.940924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.940900    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:45.955279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.955253    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.955735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.955308    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6g9h\" (UniqueName: \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.956089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.956065    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:45.997362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:45.997286    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:13:46.018454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.018418    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6g9h\" (UniqueName: \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\") pod \"payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:46.056274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.056235    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nz6qw\" (UniqueName: \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.056472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.056349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.065720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.065682    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:46.168743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.162016    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nz6qw\" (UniqueName: \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.168743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.162126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.168743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.163331    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.225676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.225640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nz6qw\" (UniqueName: \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\") pod \"payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.255548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.255457    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.440932 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:46.440901    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod713a6c2c_afec_41e9_afb8_7c3cbe4ac1bd.slice/crio-49d3dfe4f586861dac4bcd228a0f7670c9b80f3399671052aba3626a72562c14 WatchSource:0}: Error finding container 49d3dfe4f586861dac4bcd228a0f7670c9b80f3399671052aba3626a72562c14: Status 404 returned error can't find the container with id 49d3dfe4f586861dac4bcd228a0f7670c9b80f3399671052aba3626a72562c14
Oct 30 22:13:46.443269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.443244    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:13:46.630449 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:13:46.630408    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddbc916b9_d00a_495a_993d_2c77bb1cd2a3.slice/crio-34a3eca544a7e277d824a5b3265d432e2c608e8805ec7f2679c28de6b825fb7e WatchSource:0}: Error finding container 34a3eca544a7e277d824a5b3265d432e2c608e8805ec7f2679c28de6b825fb7e: Status 404 returned error can't find the container with id 34a3eca544a7e277d824a5b3265d432e2c608e8805ec7f2679c28de6b825fb7e
Oct 30 22:13:46.632963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.632935    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:13:46.739240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.739200    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" event={"ID":"dbc916b9-d00a-495a-993d-2c77bb1cd2a3","Type":"ContainerStarted","Data":"8642a72afdb6d080255d24346944885535d6c12b51ca0625bbffb623e5c21736"}
Oct 30 22:13:46.739409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.739275    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" event={"ID":"dbc916b9-d00a-495a-993d-2c77bb1cd2a3","Type":"ContainerStarted","Data":"34a3eca544a7e277d824a5b3265d432e2c608e8805ec7f2679c28de6b825fb7e"}
Oct 30 22:13:46.739484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.739439    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:46.741340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.741316    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" event={"ID":"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd","Type":"ContainerStarted","Data":"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"}
Oct 30 22:13:46.741490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.741366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" event={"ID":"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd","Type":"ContainerStarted","Data":"49d3dfe4f586861dac4bcd228a0f7670c9b80f3399671052aba3626a72562c14"}
Oct 30 22:13:46.741490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.741397    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:46.860279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.860227    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" podStartSLOduration=1.8602101979999999 podStartE2EDuration="1.860210198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:46.856952185 +0000 UTC m=+2150.030276614" watchObservedRunningTime="2026-10-30 22:13:46.860210198 +0000 UTC m=+2150.033534626"
Oct 30 22:13:46.975716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:46.975667    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" podStartSLOduration=1.9756515289999999 podStartE2EDuration="1.975651529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:13:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:13:46.974038711 +0000 UTC m=+2150.147363161" watchObservedRunningTime="2026-10-30 22:13:46.975651529 +0000 UTC m=+2150.148976001"
Oct 30 22:13:49.678665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:49.678630    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:13:49.679135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:49.678701    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:13:50.809695 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:13:50.809654    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:13:51.695615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:51.695557    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:13:57.752096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:57.752061    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:13:57.752651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:57.752157    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:13:58.326335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:58.326298    4882 scope.go:122] "RemoveContainer" containerID="36fbb4f71d4e770428fbfcaaf42b79748c215453de58a7f1eb5c8fa537b45c6d"
Oct 30 22:13:58.337870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:58.337833    4882 scope.go:122] "RemoveContainer" containerID="7e18972bdfef49747c7736676693a37acaf07d6de4475c752cd07da339c039e5"
Oct 30 22:13:58.347858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:13:58.347831    4882 scope.go:122] "RemoveContainer" containerID="c8b9dadb5e6dfd15344788479a4b9360f873bcf8e02d187721c36efb6cd8d67b"
Oct 30 22:14:04.133644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:04.133576    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"]
Oct 30 22:14:04.183454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:04.183415    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-52f191-f9xwn"]
Oct 30 22:14:04.943617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:04.943549    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4aff6c04-5e56-4a82-a412-993b7551f311" path="/var/lib/kubelet/pods/4aff6c04-5e56-4a82-a412-993b7551f311/volumes"
Oct 30 22:14:07.049685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.049601    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:14:07.055537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.055513    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.108463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.108427    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:14:07.120319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.120282    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.120536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.120513    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4m46\" (UniqueName: \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.155137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.155105    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fe745d\""
Oct 30 22:14:07.155354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.155240    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fe745d-dockercfg-qmqw9\""
Oct 30 22:14:07.211094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.211055    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:14:07.217210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.217161    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.222465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.222425    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4m46\" (UniqueName: \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.222652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.222563    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.223518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.223490    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.253201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.253165    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:14:07.285142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.285101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4m46\" (UniqueName: \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.323796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.323718    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.323796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.323761    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psp5v\" (UniqueName: \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.370082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.370046    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.425993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.425296    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.425993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.425358    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psp5v\" (UniqueName: \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.426671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.426507    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.487950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.487912    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psp5v\" (UniqueName: \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\") pod \"payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.535072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.535026    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:07.744422 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:14:07.744391    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a5c22a7_1601_431e_b7cf_1f140bcd0066.slice/crio-a5ab06daecb54d92c696b026509e7d829bf0c8a93954c9ddcc9a6ceb06223fef WatchSource:0}: Error finding container a5ab06daecb54d92c696b026509e7d829bf0c8a93954c9ddcc9a6ceb06223fef: Status 404 returned error can't find the container with id a5ab06daecb54d92c696b026509e7d829bf0c8a93954c9ddcc9a6ceb06223fef
Oct 30 22:14:07.747535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.747510    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:14:07.882138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.882056    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" event={"ID":"9a5c22a7-1601-431e-b7cf-1f140bcd0066","Type":"ContainerStarted","Data":"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"}
Oct 30 22:14:07.882138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.882106    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" event={"ID":"9a5c22a7-1601-431e-b7cf-1f140bcd0066","Type":"ContainerStarted","Data":"a5ab06daecb54d92c696b026509e7d829bf0c8a93954c9ddcc9a6ceb06223fef"}
Oct 30 22:14:07.882334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.882203    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:07.916006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:07.915970    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:14:08.000533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:08.000478    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" podStartSLOduration=1.00046313 podStartE2EDuration="1.00046313s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:07.998490519 +0000 UTC m=+2171.171814945" watchObservedRunningTime="2026-10-30 22:14:08.00046313 +0000 UTC m=+2171.173787557"
Oct 30 22:14:08.890771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:08.890737    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" event={"ID":"1166b5f1-3b27-4c4e-a837-ad17bd1822b2","Type":"ContainerStarted","Data":"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"}
Oct 30 22:14:08.891212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:08.890792    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" event={"ID":"1166b5f1-3b27-4c4e-a837-ad17bd1822b2","Type":"ContainerStarted","Data":"18b3bbc61ca400baeb2d3ff12a179737cc352440a2eae735fb44ca74274aa0e4"}
Oct 30 22:14:08.891212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:08.890988    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:09.008574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:09.008510    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" podStartSLOduration=2.008496212 podStartE2EDuration="2.008496212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:09.007437785 +0000 UTC m=+2172.180762212" watchObservedRunningTime="2026-10-30 22:14:09.008496212 +0000 UTC m=+2172.181820673"
Oct 30 22:14:12.103872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.103834    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"]
Oct 30 22:14:12.108369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.108350    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.135248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.135217    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"]
Oct 30 22:14:12.162061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.162030    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:14:12.162457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.162432    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:14:12.181078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.181044    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.181276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.181253    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.181320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.181308    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5bk4\" (UniqueName: \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.282535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.282499    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.282753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.282559    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5bk4\" (UniqueName: \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.282753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.282650    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.283317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.283286    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.284963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.284942    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.346746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.346718    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5bk4\" (UniqueName: \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\") pod \"maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.422865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.422780    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:12.796075 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:14:12.796038    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cacf374_f0be_4fdb_8200_d770a322bfd1.slice/crio-020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127 WatchSource:0}: Error finding container 020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127: Status 404 returned error can't find the container with id 020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127
Oct 30 22:14:12.799347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.799315    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"]
Oct 30 22:14:12.919331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:12.919291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv" event={"ID":"4cacf374-f0be-4fdb-8200-d770a322bfd1","Type":"ContainerStarted","Data":"020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127"}
Oct 30 22:14:13.928907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:13.928874    4882 generic.go:356] "Generic (PLEG): container finished" podID="4cacf374-f0be-4fdb-8200-d770a322bfd1" containerID="5788e7a1515a9504ea3680a99f07456f653fe81a47dbc1ae79192f3203e31df5" exitCode=0
Oct 30 22:14:13.929289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:13.928953    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv" event={"ID":"4cacf374-f0be-4fdb-8200-d770a322bfd1","Type":"ContainerDied","Data":"5788e7a1515a9504ea3680a99f07456f653fe81a47dbc1ae79192f3203e31df5"}
Oct 30 22:14:15.104417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.104382    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:15.229561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.229478    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\") pod \"4cacf374-f0be-4fdb-8200-d770a322bfd1\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") "
Oct 30 22:14:15.229795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.229774    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\" (UniqueName: \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\") pod \"4cacf374-f0be-4fdb-8200-d770a322bfd1\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") "
Oct 30 22:14:15.229875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.229850    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\") pod \"4cacf374-f0be-4fdb-8200-d770a322bfd1\" (UID: \"4cacf374-f0be-4fdb-8200-d770a322bfd1\") "
Oct 30 22:14:15.229939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.229898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca" pod "4cacf374-f0be-4fdb-8200-d770a322bfd1" (UID: "4cacf374-f0be-4fdb-8200-d770a322bfd1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:15.230269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.230248    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.231985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.231958    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4" pod "4cacf374-f0be-4fdb-8200-d770a322bfd1" (UID: "4cacf374-f0be-4fdb-8200-d770a322bfd1"). InnerVolumeSpecName "kube-api-access-x5bk4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:15.232092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.232005    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca" pod "4cacf374-f0be-4fdb-8200-d770a322bfd1" (UID: "4cacf374-f0be-4fdb-8200-d770a322bfd1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:14:15.331088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.331052    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5bk4\" (UniqueName: \"kubernetes.io/projected/4cacf374-f0be-4fdb-8200-d770a322bfd1-kube-api-access-x5bk4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.331088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.331082    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4cacf374-f0be-4fdb-8200-d770a322bfd1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:15.942445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.942418    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"
Oct 30 22:14:15.942445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.942429    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv" event={"ID":"4cacf374-f0be-4fdb-8200-d770a322bfd1","Type":"ContainerDied","Data":"020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127"}
Oct 30 22:14:15.942679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:15.942481    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="020dfa4e1082e0fa5bf3159a007f81bdb01624bbad312f5a2435afbe59c02127"
Oct 30 22:14:18.894452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:18.894423    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:14:19.444716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.444682    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:14:19.445357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.445341    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4cacf374-f0be-4fdb-8200-d770a322bfd1" containerName="revoke-keys"
Oct 30 22:14:19.445412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.445359    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cacf374-f0be-4fdb-8200-d770a322bfd1" containerName="revoke-keys"
Oct 30 22:14:19.445465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.445456    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cacf374-f0be-4fdb-8200-d770a322bfd1" containerName="revoke-keys"
Oct 30 22:14:19.449153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.449136    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.504119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.504080    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:14:19.550684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.550643    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-a4257d\""
Oct 30 22:14:19.550684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.550678    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-a4257d-dockercfg-ql9l2\""
Oct 30 22:14:19.583392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.583363    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.583532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.583417    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jtk\" (UniqueName: \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.684649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.684563    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.684833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.684670    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jtk\" (UniqueName: \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.685271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.685249    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.748917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.748846    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jtk\" (UniqueName: \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\") pod \"payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.762938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.762903    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:19.900125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:19.900093    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:14:20.142769 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:14:20.142716    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacf9268b_77ed_491b_bb2e_ab0f95f64b53.slice/crio-abc71de0916f4e2d1461baeab457ec5fe2c5b6de2c8f5dfdebba01acb211be00 WatchSource:0}: Error finding container abc71de0916f4e2d1461baeab457ec5fe2c5b6de2c8f5dfdebba01acb211be00: Status 404 returned error can't find the container with id abc71de0916f4e2d1461baeab457ec5fe2c5b6de2c8f5dfdebba01acb211be00
Oct 30 22:14:20.145093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:20.145065    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:14:20.809674 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:14:20.809637    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:14:20.976158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:20.976126    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" event={"ID":"acf9268b-77ed-491b-bb2e-ab0f95f64b53","Type":"ContainerStarted","Data":"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"}
Oct 30 22:14:20.976599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:20.976177    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" event={"ID":"acf9268b-77ed-491b-bb2e-ab0f95f64b53","Type":"ContainerStarted","Data":"abc71de0916f4e2d1461baeab457ec5fe2c5b6de2c8f5dfdebba01acb211be00"}
Oct 30 22:14:20.976599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:20.976199    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:21.094798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:21.094732    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" podStartSLOduration=2.094712075 podStartE2EDuration="2.094712075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:21.093324524 +0000 UTC m=+2184.266648952" watchObservedRunningTime="2026-10-30 22:14:21.094712075 +0000 UTC m=+2184.268036505"
Oct 30 22:14:23.581934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:23.581895    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:14:23.691023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:23.690988    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:14:28.582767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.582716    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" podUID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" containerName="payload-processing" containerID="cri-o://fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149" gracePeriod=25
Oct 30 22:14:28.692390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.692310    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" podUID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" containerName="payload-pre-processing" containerID="cri-o://e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9" gracePeriod=25
Oct 30 22:14:28.866537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.866509    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:14:28.898506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.898474    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\") pod \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") "
Oct 30 22:14:28.898690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.898672    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\" (UniqueName: \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\") pod \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\" (UID: \"aaf4a5f9-1e1f-49d2-b529-be47062ece43\") "
Oct 30 22:14:28.898965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.898943    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume" pod "aaf4a5f9-1e1f-49d2-b529-be47062ece43" (UID: "aaf4a5f9-1e1f-49d2-b529-be47062ece43"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:28.899092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.899078    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaf4a5f9-1e1f-49d2-b529-be47062ece43-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:28.900874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.900845    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm" pod "aaf4a5f9-1e1f-49d2-b529-be47062ece43" (UID: "aaf4a5f9-1e1f-49d2-b529-be47062ece43"). InnerVolumeSpecName "kube-api-access-rswkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:28.963900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:28.963872    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:14:29.000204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.000165    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rswkm\" (UniqueName: \"kubernetes.io/projected/aaf4a5f9-1e1f-49d2-b529-be47062ece43-kube-api-access-rswkm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:29.030387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.030352    4882 generic.go:356] "Generic (PLEG): container finished" podID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" containerID="fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149" exitCode=0
Oct 30 22:14:29.030580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.030410    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"
Oct 30 22:14:29.030580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.030447    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" event={"ID":"aaf4a5f9-1e1f-49d2-b529-be47062ece43","Type":"ContainerDied","Data":"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"}
Oct 30 22:14:29.030580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.030516    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs" event={"ID":"aaf4a5f9-1e1f-49d2-b529-be47062ece43","Type":"ContainerDied","Data":"4e212d11900acdd1aafc945c9fdb9540d17d6e00de43c45b81b8482dddc3cbeb"}
Oct 30 22:14:29.030580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.030547    4882 scope.go:122] "RemoveContainer" containerID="fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"
Oct 30 22:14:29.032010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.031981    4882 generic.go:356] "Generic (PLEG): container finished" podID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" containerID="e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9" exitCode=0
Oct 30 22:14:29.032138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.032023    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"
Oct 30 22:14:29.032138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.032072    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" event={"ID":"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a","Type":"ContainerDied","Data":"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"}
Oct 30 22:14:29.032234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.032148    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m" event={"ID":"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a","Type":"ContainerDied","Data":"c002cfe977b3fcecdb3fa30dc9086176eb05c6b7fc8657d405b4ce25aceff643"}
Oct 30 22:14:29.040926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.040902    4882 scope.go:122] "RemoveContainer" containerID="fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"
Oct 30 22:14:29.041316 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:14:29.041286    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149\": container with ID starting with fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149 not found: ID does not exist" containerID="fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"
Oct 30 22:14:29.041367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.041329    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149"} err="failed to get container status \"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149\": rpc error: code = NotFound desc = could not find container \"fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149\": container with ID starting with fbca20dc8f5c047cdb87553eacec88fd78be58ae339b338240abbc9a442b1149 not found: ID does not exist"
Oct 30 22:14:29.041367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.041347    4882 scope.go:122] "RemoveContainer" containerID="e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"
Oct 30 22:14:29.050875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.050855    4882 scope.go:122] "RemoveContainer" containerID="e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"
Oct 30 22:14:29.051208 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:14:29.051185    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9\": container with ID starting with e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9 not found: ID does not exist" containerID="e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"
Oct 30 22:14:29.051317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.051214    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9"} err="failed to get container status \"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9\": rpc error: code = NotFound desc = could not find container \"e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9\": container with ID starting with e5b81b1ec392781f1c7eb0cfb59d077c563e7e92f12f5d88928ed8d4f8aacfb9 not found: ID does not exist"
Oct 30 22:14:29.101240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.101155    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\") pod \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") "
Oct 30 22:14:29.101405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.101389    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\" (UniqueName: \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\") pod \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\" (UID: \"cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a\") "
Oct 30 22:14:29.101625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.101573    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume" pod "cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" (UID: "cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:14:29.101883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.101864    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:29.105037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.105008    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn" pod "cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" (UID: "cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a"). InnerVolumeSpecName "kube-api-access-znnnn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:14:29.202541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.202497    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnnn\" (UniqueName: \"kubernetes.io/projected/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a-kube-api-access-znnnn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:14:29.280288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.280246    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:14:29.283295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.283256    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-a3aa65-544599d96c-pf4fs"]
Oct 30 22:14:29.456036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.455947    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:14:29.458990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:29.458962    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-a3aa65-6bc6fb5f5f-m7k7m"]
Oct 30 22:14:30.944047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:30.944013    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" path="/var/lib/kubelet/pods/aaf4a5f9-1e1f-49d2-b529-be47062ece43/volumes"
Oct 30 22:14:30.944535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:30.944515    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" path="/var/lib/kubelet/pods/cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a/volumes"
Oct 30 22:14:31.985662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:31.985622    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:14:35.932040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932004    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:14:35.932693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932677    4882 cpu_manager.go:414] "containerMap: removing container" podUID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" containerName="payload-processing"
Oct 30 22:14:35.932748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932694    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" containerName="payload-processing"
Oct 30 22:14:35.932781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932753    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" containerName="payload-pre-processing"
Oct 30 22:14:35.932781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932759    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" containerName="payload-pre-processing"
Oct 30 22:14:35.932887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932878    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="aaf4a5f9-1e1f-49d2-b529-be47062ece43" containerName="payload-processing"
Oct 30 22:14:35.932924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.932895    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd95da3c-42a3-4d35-9cb4-2e03c0f6c01a" containerName="payload-pre-processing"
Oct 30 22:14:35.938196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.938170    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:35.980574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.980531    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:35.980788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.980645    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hr67\" (UniqueName: \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:35.991365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:35.991317    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:14:36.038166 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.038137    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-95b84c\""
Oct 30 22:14:36.038341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.038246    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-95b84c-dockercfg-qkksf\""
Oct 30 22:14:36.081845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.081811    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:36.082016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.081910    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hr67\" (UniqueName: \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:36.082625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.082581    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:36.143920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.143885    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hr67\" (UniqueName: \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\") pod \"payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:36.252851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.252757    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:36.628523 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:14:36.628489    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e124c0b_7950_436d_a84b_ca91bd1babf2.slice/crio-a2fe1f629ae332c66795bf9509626b3b35ddae882cba9eb56deef71c5e6f9b47 WatchSource:0}: Error finding container a2fe1f629ae332c66795bf9509626b3b35ddae882cba9eb56deef71c5e6f9b47: Status 404 returned error can't find the container with id a2fe1f629ae332c66795bf9509626b3b35ddae882cba9eb56deef71c5e6f9b47
Oct 30 22:14:36.632709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.632681    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:14:36.728647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.728573    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:14:36.733069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.733046    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.791154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.791114    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.791477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.791449    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwkwp\" (UniqueName: \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.795407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.795377    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:14:36.893105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.893013    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwkwp\" (UniqueName: \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.893105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.893077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.893903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.893882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:36.954441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:36.954408    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwkwp\" (UniqueName: \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\") pod \"payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:37.048967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.048934    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:37.097118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.097085    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" event={"ID":"3e124c0b-7950-436d-a84b-ca91bd1babf2","Type":"ContainerStarted","Data":"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"}
Oct 30 22:14:37.097309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.097147    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" event={"ID":"3e124c0b-7950-436d-a84b-ca91bd1babf2","Type":"ContainerStarted","Data":"a2fe1f629ae332c66795bf9509626b3b35ddae882cba9eb56deef71c5e6f9b47"}
Oct 30 22:14:37.097309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.097180    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:37.218019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.217911    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" podStartSLOduration=2.217893158 podStartE2EDuration="2.217893158s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:37.214272604 +0000 UTC m=+2200.387597031" watchObservedRunningTime="2026-10-30 22:14:37.217893158 +0000 UTC m=+2200.391217582"
Oct 30 22:14:37.426626 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:14:37.426391    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20c77d7d_7635_4a9f_972a_2c7b8533ae16.slice/crio-5b8466076653ce0622ff21d065163b7542928d095931a5c640c4e1283505b4f6 WatchSource:0}: Error finding container 5b8466076653ce0622ff21d065163b7542928d095931a5c640c4e1283505b4f6: Status 404 returned error can't find the container with id 5b8466076653ce0622ff21d065163b7542928d095931a5c640c4e1283505b4f6
Oct 30 22:14:37.430062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:37.430028    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:14:38.106258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:38.106221    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" event={"ID":"20c77d7d-7635-4a9f-972a-2c7b8533ae16","Type":"ContainerStarted","Data":"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"}
Oct 30 22:14:38.106798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:38.106276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" event={"ID":"20c77d7d-7635-4a9f-972a-2c7b8533ae16","Type":"ContainerStarted","Data":"5b8466076653ce0622ff21d065163b7542928d095931a5c640c4e1283505b4f6"}
Oct 30 22:14:38.106798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:38.106492    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:38.228317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:38.228259    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" podStartSLOduration=2.228242551 podStartE2EDuration="2.228242551s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:14:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:14:38.225655785 +0000 UTC m=+2201.398980212" watchObservedRunningTime="2026-10-30 22:14:38.228242551 +0000 UTC m=+2201.401567009"
Oct 30 22:14:40.143705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:40.143673    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:14:48.109013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:48.108977    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:14:49.117532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:49.117501    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:14:50.807627 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:14:50.807574    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:14:58.413162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:58.413119    4882 scope.go:122] "RemoveContainer" containerID="8418e6d951d0bf275db884977f00d1cf1d2aa79e211111f968361a2a078e1249"
Oct 30 22:14:58.423408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:14:58.423386    4882 scope.go:122] "RemoveContainer" containerID="2a327e98023f865113b148b367781e6a766505701e4332dc5cd30af167d45662"
Oct 30 22:15:00.225642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.225600    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:00.231452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.231426    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:00.231636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.231565    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.235622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.235572    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.258088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.258058    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:00.262179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.262160    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.273797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.273761    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnb8\" (UniqueName: \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.273996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.273970    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.274073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.274013    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skjwd\" (UniqueName: \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.274073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.274042    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.274150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.274110    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngvt8\" (UniqueName: \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.274254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.274234    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.275928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.275906    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:00.280062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.280044    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.285732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.285707    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:00.290048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.290022    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.297550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.297526    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:15:00.319678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.319570    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:00.324225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.324194    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.326871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.326832    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:00.331552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.331526    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.342938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.341423    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:00.353185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.352204    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:00.355761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.355738    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:00.366247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.366213    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:00.383810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.383772    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.384017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.383867    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.384111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.383901    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkbng\" (UniqueName: \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.384202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384184    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.384287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384270    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.384345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384327    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnb8\" (UniqueName: \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.384569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384551    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.384687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384666    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.384733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384705    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrfc5\" (UniqueName: \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.384898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384870    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.384949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384932    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skjwd\" (UniqueName: \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.384983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384959    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.384983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.384977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngvt8\" (UniqueName: \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.385048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.385024    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwnrf\" (UniqueName: \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.385048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.385044    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9x6b\" (UniqueName: \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.385507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.385489    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.386191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.386162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.387927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.387805    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:00.387927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.387863    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:00.388120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.387962    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.393741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.390432    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:00.396629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.395786    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:00.396779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.396708    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.401957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.401928    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.403158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.403136    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:00.421420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.421387    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:00.437153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.436870    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:00.452208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.452176    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:00.454910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.454882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnb8\" (UniqueName: \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.459348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.459314    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skjwd\" (UniqueName: \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.459495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.459325    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngvt8\" (UniqueName: \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\") pod \"maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.486393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486315    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.486393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486354    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrfc5\" (UniqueName: \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.486662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486465    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.486662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486504    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.486662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486565    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgqvc\" (UniqueName: \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.486861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486749    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hljs\" (UniqueName: \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.486861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486810    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.486967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486908    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwnrf\" (UniqueName: \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.486967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.486949    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9x6b\" (UniqueName: \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.487081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.487136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487079    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llzt4\" (UniqueName: \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.487192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487156    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.487351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487328    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkbng\" (UniqueName: \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.487442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487425    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.487502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487472    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.487849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.487824    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.488308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.488273    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.488896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.488873    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.492288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.492263    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:00.550528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.550487    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:00.557206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.557179    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:00.560242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.560206    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrfc5\" (UniqueName: \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\") pod \"maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.560381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.560268    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9x6b\" (UniqueName: \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\") pod \"maas-api-key-cleanup-29889975-knk8n\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.560428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.560407    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkbng\" (UniqueName: \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\") pod \"maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.560709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.560682    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwnrf\" (UniqueName: \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\") pod \"maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.577108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.577081    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:00.589146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589103    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.589303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589153    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.589303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589178    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgqvc\" (UniqueName: \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.589303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589207    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hljs\" (UniqueName: \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.589303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.589525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.589311    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llzt4\" (UniqueName: \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.590715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.590619    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.590715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.590641    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.590715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.590650    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.597641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.596552    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:00.608658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.606439    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:00.649101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.647838    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:00.660462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.657191    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:00.688161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.688120    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-llzt4\" (UniqueName: \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\") pod \"maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.691008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.689928    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgqvc\" (UniqueName: \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\") pod \"maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.695001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.694543    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hljs\" (UniqueName: \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.708397 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.708137    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:00.723221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.722980    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:00.733252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.732052    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:00.968161 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:00.968131    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b01b4df_6b9c_4d13_a74d_b9731cf66cec.slice/crio-28ff8013e90f7ef9476a793d616a66cf6f27ec60fbb47c7d3d8106033c8a83c9 WatchSource:0}: Error finding container 28ff8013e90f7ef9476a793d616a66cf6f27ec60fbb47c7d3d8106033c8a83c9: Status 404 returned error can't find the container with id 28ff8013e90f7ef9476a793d616a66cf6f27ec60fbb47c7d3d8106033c8a83c9
Oct 30 22:15:00.969033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:00.969007    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:01.168132 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.168094    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44e73d51_1357_476b_a4c7_29f3c0a121c9.slice/crio-6bacdd33b2fc63d4365b9781287ffc93684516d1da70436f6e070894c113806d WatchSource:0}: Error finding container 6bacdd33b2fc63d4365b9781287ffc93684516d1da70436f6e070894c113806d: Status 404 returned error can't find the container with id 6bacdd33b2fc63d4365b9781287ffc93684516d1da70436f6e070894c113806d
Oct 30 22:15:01.171956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.171932    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:01.199643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.199616    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:01.199867 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.199844    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac7e81ef_6600_4b3b_870d_8e8d828d15a1.slice/crio-b4d5321b98518e9dab655ce6da788f7c3ae3c1e3bfea06f385eae2830c08ecf4 WatchSource:0}: Error finding container b4d5321b98518e9dab655ce6da788f7c3ae3c1e3bfea06f385eae2830c08ecf4: Status 404 returned error can't find the container with id b4d5321b98518e9dab655ce6da788f7c3ae3c1e3bfea06f385eae2830c08ecf4
Oct 30 22:15:01.235890 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.234909    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c5569be_cfec_43bc_ac28_538fc7de0f11.slice/crio-94f526631376e6c52c922fb5155723eb221aafc3af696cf67e269a9f6d47731d WatchSource:0}: Error finding container 94f526631376e6c52c922fb5155723eb221aafc3af696cf67e269a9f6d47731d: Status 404 returned error can't find the container with id 94f526631376e6c52c922fb5155723eb221aafc3af696cf67e269a9f6d47731d
Oct 30 22:15:01.237726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.237696    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:01.247428 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.247395    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54245ea6_5e26_4b20_b2d8_870dbdb1a180.slice/crio-c01a8fc46f542c5a6fa706c8ba0ee2ceea8192a871eb1e838c3d12ebd200834e WatchSource:0}: Error finding container c01a8fc46f542c5a6fa706c8ba0ee2ceea8192a871eb1e838c3d12ebd200834e: Status 404 returned error can't find the container with id c01a8fc46f542c5a6fa706c8ba0ee2ceea8192a871eb1e838c3d12ebd200834e
Oct 30 22:15:01.248180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.248155    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:01.250072 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.250050    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29fdc1d3_ba80_461d_aa21_2db22628dc33.slice/crio-427a119446d8bdd557400f73bdab84846372f193d6d5946dd936905ca53f5724 WatchSource:0}: Error finding container 427a119446d8bdd557400f73bdab84846372f193d6d5946dd936905ca53f5724: Status 404 returned error can't find the container with id 427a119446d8bdd557400f73bdab84846372f193d6d5946dd936905ca53f5724
Oct 30 22:15:01.252704 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.252635    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:01.261929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.261894    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerStarted","Data":"94f526631376e6c52c922fb5155723eb221aafc3af696cf67e269a9f6d47731d"}
Oct 30 22:15:01.263565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.263540    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerStarted","Data":"c01a8fc46f542c5a6fa706c8ba0ee2ceea8192a871eb1e838c3d12ebd200834e"}
Oct 30 22:15:01.265142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.265114    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerStarted","Data":"28ff8013e90f7ef9476a793d616a66cf6f27ec60fbb47c7d3d8106033c8a83c9"}
Oct 30 22:15:01.266652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.266628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerStarted","Data":"427a119446d8bdd557400f73bdab84846372f193d6d5946dd936905ca53f5724"}
Oct 30 22:15:01.268032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.268003    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerStarted","Data":"6bacdd33b2fc63d4365b9781287ffc93684516d1da70436f6e070894c113806d"}
Oct 30 22:15:01.269422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.269399    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerStarted","Data":"b4d5321b98518e9dab655ce6da788f7c3ae3c1e3bfea06f385eae2830c08ecf4"}
Oct 30 22:15:01.355460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.355434    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:01.359801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.359773    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:01.362973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.362952    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:01.389307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:01.389281    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:01.434650 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.433698    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2251a9a9_4960_4128_b081_c22ff5f560bc.slice/crio-e6d60ed076e8cb76e37a9f3900570135806e0a9423379bbaf8c20a3f54541963 WatchSource:0}: Error finding container e6d60ed076e8cb76e37a9f3900570135806e0a9423379bbaf8c20a3f54541963: Status 404 returned error can't find the container with id e6d60ed076e8cb76e37a9f3900570135806e0a9423379bbaf8c20a3f54541963
Oct 30 22:15:01.529063 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.528978    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2f96be1_7f98_4480_9459_b3411fa98c5c.slice/crio-8817ce8fc31445b1756b3c3fae1e4517f7e14af97afdf4af182ad96010e84985 WatchSource:0}: Error finding container 8817ce8fc31445b1756b3c3fae1e4517f7e14af97afdf4af182ad96010e84985: Status 404 returned error can't find the container with id 8817ce8fc31445b1756b3c3fae1e4517f7e14af97afdf4af182ad96010e84985
Oct 30 22:15:01.530364 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:15:01.530334    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad3c024c_9c76_4caa_9136_44d3eebaf342.slice/crio-47ba3ac98dbb73bf205b587e4f00db9e4de0ac17d268d7647af773e19985ec71 WatchSource:0}: Error finding container 47ba3ac98dbb73bf205b587e4f00db9e4de0ac17d268d7647af773e19985ec71: Status 404 returned error can't find the container with id 47ba3ac98dbb73bf205b587e4f00db9e4de0ac17d268d7647af773e19985ec71
Oct 30 22:15:02.278629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.278560    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerStarted","Data":"9cdbedd427339887f70ba6fc20ab27a68572093ef77958936871a4e705132421"}
Oct 30 22:15:02.280312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.280283    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerStarted","Data":"5ae34bddd5476b1a2dc1e2af62c35bb91c5b4540d54f48eec1fd2f09e882f6b0"}
Oct 30 22:15:02.280444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.280325    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerStarted","Data":"e6d60ed076e8cb76e37a9f3900570135806e0a9423379bbaf8c20a3f54541963"}
Oct 30 22:15:02.281833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.281804    4882 generic.go:356] "Generic (PLEG): container finished" podID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerID="60734f9aafc746be551106de65d3ae47d7bc8ff5e54a16576c4397b3b813bc34" exitCode=7
Oct 30 22:15:02.281952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.281879    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerDied","Data":"60734f9aafc746be551106de65d3ae47d7bc8ff5e54a16576c4397b3b813bc34"}
Oct 30 22:15:02.282141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.282120    4882 scope.go:122] "RemoveContainer" containerID="60734f9aafc746be551106de65d3ae47d7bc8ff5e54a16576c4397b3b813bc34"
Oct 30 22:15:02.283641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.283617    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerStarted","Data":"a5e43ebf0e5c5d7f53e5838668ec4ba2fe2e4389578f072b1b8cc5b6786884e9"}
Oct 30 22:15:02.284838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.284815    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerStarted","Data":"8817ce8fc31445b1756b3c3fae1e4517f7e14af97afdf4af182ad96010e84985"}
Oct 30 22:15:02.286184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.286159    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerStarted","Data":"d07f389dd46996257f4b328366510979e870253b071b082c4a11cefe2d454da3"}
Oct 30 22:15:02.287689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.287667    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerStarted","Data":"40220b7f8e18e786675bf1cb9188718dd7179ad1466d3ea0ec59fcdcae8cab9e"}
Oct 30 22:15:02.289240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.289216    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerStarted","Data":"735e38e17ac0e78df826266f377d2886c71ace06ff3208d568e7e07d511f341d"}
Oct 30 22:15:02.290838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.290817    4882 generic.go:356] "Generic (PLEG): container finished" podID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerID="c26775c2a1e184437a19519272eac8b6b75aa1d16adea26466970ca7e43e13ef" exitCode=60
Oct 30 22:15:02.290934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.290878    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerDied","Data":"c26775c2a1e184437a19519272eac8b6b75aa1d16adea26466970ca7e43e13ef"}
Oct 30 22:15:02.291062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.291042    4882 scope.go:122] "RemoveContainer" containerID="c26775c2a1e184437a19519272eac8b6b75aa1d16adea26466970ca7e43e13ef"
Oct 30 22:15:02.292125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.292107    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerStarted","Data":"47ba3ac98dbb73bf205b587e4f00db9e4de0ac17d268d7647af773e19985ec71"}
Oct 30 22:15:02.422837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.422781    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" podStartSLOduration=2.422768178 podStartE2EDuration="2.422768178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.421509132 +0000 UTC m=+2225.594833559" watchObservedRunningTime="2026-10-30 22:15:02.422768178 +0000 UTC m=+2225.596092602"
Oct 30 22:15:02.553520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.553465    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" podStartSLOduration=2.553451689 podStartE2EDuration="2.553451689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.551099361 +0000 UTC m=+2225.724423806" watchObservedRunningTime="2026-10-30 22:15:02.553451689 +0000 UTC m=+2225.726776116"
Oct 30 22:15:02.675245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:02.675185    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" podStartSLOduration=2.675170489 podStartE2EDuration="2.675170489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:02.674001116 +0000 UTC m=+2225.847325542" watchObservedRunningTime="2026-10-30 22:15:02.675170489 +0000 UTC m=+2225.848494953"
Oct 30 22:15:03.050239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.050186    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" podStartSLOduration=3.050170709 podStartE2EDuration="3.050170709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.0480156 +0000 UTC m=+2226.221340026" watchObservedRunningTime="2026-10-30 22:15:03.050170709 +0000 UTC m=+2226.223495136"
Oct 30 22:15:03.183086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.182984    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" podStartSLOduration=3.182970943 podStartE2EDuration="3.182970943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.180465811 +0000 UTC m=+2226.353790238" watchObservedRunningTime="2026-10-30 22:15:03.182970943 +0000 UTC m=+2226.356295383"
Oct 30 22:15:03.300893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.300853    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerStarted","Data":"8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"}
Oct 30 22:15:03.302596 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.302561    4882 generic.go:356] "Generic (PLEG): container finished" podID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerID="9c43b8db994f6aa3ac0dce2bc7143e4d5bd9d6bf0a7b49bcca6c96cf0bc354cf" exitCode=60
Oct 30 22:15:03.302719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.302648    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerDied","Data":"9c43b8db994f6aa3ac0dce2bc7143e4d5bd9d6bf0a7b49bcca6c96cf0bc354cf"}
Oct 30 22:15:03.302921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.302903    4882 scope.go:122] "RemoveContainer" containerID="9c43b8db994f6aa3ac0dce2bc7143e4d5bd9d6bf0a7b49bcca6c96cf0bc354cf"
Oct 30 22:15:03.305047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.305026    4882 generic.go:356] "Generic (PLEG): container finished" podID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerID="9cdbedd427339887f70ba6fc20ab27a68572093ef77958936871a4e705132421" exitCode=60
Oct 30 22:15:03.305126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.305107    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerDied","Data":"9cdbedd427339887f70ba6fc20ab27a68572093ef77958936871a4e705132421"}
Oct 30 22:15:03.305330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.305312    4882 scope.go:122] "RemoveContainer" containerID="9cdbedd427339887f70ba6fc20ab27a68572093ef77958936871a4e705132421"
Oct 30 22:15:03.306955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.306931    4882 generic.go:356] "Generic (PLEG): container finished" podID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerID="5ae34bddd5476b1a2dc1e2af62c35bb91c5b4540d54f48eec1fd2f09e882f6b0" exitCode=60
Oct 30 22:15:03.307072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.307023    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerDied","Data":"5ae34bddd5476b1a2dc1e2af62c35bb91c5b4540d54f48eec1fd2f09e882f6b0"}
Oct 30 22:15:03.307236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.307220    4882 scope.go:122] "RemoveContainer" containerID="5ae34bddd5476b1a2dc1e2af62c35bb91c5b4540d54f48eec1fd2f09e882f6b0"
Oct 30 22:15:03.308994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.308969    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerStarted","Data":"883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"}
Oct 30 22:15:03.310730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.310709    4882 generic.go:356] "Generic (PLEG): container finished" podID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerID="a5e43ebf0e5c5d7f53e5838668ec4ba2fe2e4389578f072b1b8cc5b6786884e9" exitCode=60
Oct 30 22:15:03.310834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.310795    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerDied","Data":"a5e43ebf0e5c5d7f53e5838668ec4ba2fe2e4389578f072b1b8cc5b6786884e9"}
Oct 30 22:15:03.311028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.311007    4882 scope.go:122] "RemoveContainer" containerID="a5e43ebf0e5c5d7f53e5838668ec4ba2fe2e4389578f072b1b8cc5b6786884e9"
Oct 30 22:15:03.312321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.312297    4882 generic.go:356] "Generic (PLEG): container finished" podID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerID="b018861318bb81221023987fe54d54c1a5106d1f44f3702e1f5ada7f81c5b08a" exitCode=60
Oct 30 22:15:03.312407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.312374    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerDied","Data":"b018861318bb81221023987fe54d54c1a5106d1f44f3702e1f5ada7f81c5b08a"}
Oct 30 22:15:03.312561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.312548    4882 scope.go:122] "RemoveContainer" containerID="b018861318bb81221023987fe54d54c1a5106d1f44f3702e1f5ada7f81c5b08a"
Oct 30 22:15:03.314688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.314305    4882 generic.go:356] "Generic (PLEG): container finished" podID="7e58a043-0b72-4471-8694-b7480513764b" containerID="9506826b3d8af61860f49768417c9bd0d3a195e2a25638ed61fc6e63422ef472" exitCode=60
Oct 30 22:15:03.314688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.314432    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerDied","Data":"9506826b3d8af61860f49768417c9bd0d3a195e2a25638ed61fc6e63422ef472"}
Oct 30 22:15:03.314688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.314576    4882 scope.go:122] "RemoveContainer" containerID="9506826b3d8af61860f49768417c9bd0d3a195e2a25638ed61fc6e63422ef472"
Oct 30 22:15:03.316686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.316666    4882 generic.go:356] "Generic (PLEG): container finished" podID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerID="40220b7f8e18e786675bf1cb9188718dd7179ad1466d3ea0ec59fcdcae8cab9e" exitCode=60
Oct 30 22:15:03.316824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.316801    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerDied","Data":"40220b7f8e18e786675bf1cb9188718dd7179ad1466d3ea0ec59fcdcae8cab9e"}
Oct 30 22:15:03.316956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.316939    4882 scope.go:122] "RemoveContainer" containerID="40220b7f8e18e786675bf1cb9188718dd7179ad1466d3ea0ec59fcdcae8cab9e"
Oct 30 22:15:03.318533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.318513    4882 generic.go:356] "Generic (PLEG): container finished" podID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerID="735e38e17ac0e78df826266f377d2886c71ace06ff3208d568e7e07d511f341d" exitCode=60
Oct 30 22:15:03.318633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.318540    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerDied","Data":"735e38e17ac0e78df826266f377d2886c71ace06ff3208d568e7e07d511f341d"}
Oct 30 22:15:03.318743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.318730    4882 scope.go:122] "RemoveContainer" containerID="735e38e17ac0e78df826266f377d2886c71ace06ff3208d568e7e07d511f341d"
Oct 30 22:15:03.430940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.429725    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" podStartSLOduration=3.429702935 podStartE2EDuration="3.429702935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.426253263 +0000 UTC m=+2226.599577689" watchObservedRunningTime="2026-10-30 22:15:03.429702935 +0000 UTC m=+2226.603027364"
Oct 30 22:15:03.945869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:03.945813    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" podStartSLOduration=3.945798203 podStartE2EDuration="3.945798203s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:03.941371853 +0000 UTC m=+2227.114696281" watchObservedRunningTime="2026-10-30 22:15:03.945798203 +0000 UTC m=+2227.119122672"
Oct 30 22:15:04.330564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.330531    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerStarted","Data":"01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"}
Oct 30 22:15:04.332832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.332799    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerStarted","Data":"ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"}
Oct 30 22:15:04.334947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.334919    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerStarted","Data":"dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"}
Oct 30 22:15:04.336809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.336779    4882 generic.go:356] "Generic (PLEG): container finished" podID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerID="883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5" exitCode=7
Oct 30 22:15:04.336914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.336833    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerDied","Data":"883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"}
Oct 30 22:15:04.336953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.336916    4882 scope.go:122] "RemoveContainer" containerID="60734f9aafc746be551106de65d3ae47d7bc8ff5e54a16576c4397b3b813bc34"
Oct 30 22:15:04.337089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.337075    4882 scope.go:122] "RemoveContainer" containerID="883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"
Oct 30 22:15:04.337358 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:04.337341    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj_odh-ai-gateway-infra(44e73d51-1357-476b-a4c7-29f3c0a121c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9"
Oct 30 22:15:04.339352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.339319    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerStarted","Data":"a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"}
Oct 30 22:15:04.341337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.341313    4882 generic.go:356] "Generic (PLEG): container finished" podID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerID="86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93" exitCode=60
Oct 30 22:15:04.341439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.341395    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerDied","Data":"86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93"}
Oct 30 22:15:04.341678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.341657    4882 scope.go:122] "RemoveContainer" containerID="86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93"
Oct 30 22:15:04.341953 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:04.341928    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r_odh-ai-gateway-infra(e2f96be1-7f98-4480-9459-b3411fa98c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c"
Oct 30 22:15:04.343457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.343426    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerStarted","Data":"46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"}
Oct 30 22:15:04.345754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.345726    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerStarted","Data":"234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"}
Oct 30 22:15:04.347883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.347863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerStarted","Data":"43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"}
Oct 30 22:15:04.349996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.349975    4882 generic.go:356] "Generic (PLEG): container finished" podID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerID="8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834" exitCode=60
Oct 30 22:15:04.350093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.350011    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerDied","Data":"8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"}
Oct 30 22:15:04.350230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.350216    4882 scope.go:122] "RemoveContainer" containerID="8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"
Oct 30 22:15:04.350446 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:04.350427    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz_odh-ai-gateway-infra(1b01b4df-6b9c-4d13-a74d-b9731cf66cec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec"
Oct 30 22:15:04.351656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.351638    4882 scope.go:122] "RemoveContainer" containerID="b018861318bb81221023987fe54d54c1a5106d1f44f3702e1f5ada7f81c5b08a"
Oct 30 22:15:04.365488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.365465    4882 scope.go:122] "RemoveContainer" containerID="c26775c2a1e184437a19519272eac8b6b75aa1d16adea26466970ca7e43e13ef"
Oct 30 22:15:04.800686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:04.800568    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" podStartSLOduration=4.800550119 podStartE2EDuration="4.800550119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:04.795990639 +0000 UTC m=+2227.969315077" watchObservedRunningTime="2026-10-30 22:15:04.800550119 +0000 UTC m=+2227.973874548"
Oct 30 22:15:05.359613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.359556    4882 generic.go:356] "Generic (PLEG): container finished" podID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerID="43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6" exitCode=60
Oct 30 22:15:05.360065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.359663    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerDied","Data":"43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"}
Oct 30 22:15:05.360065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.359725    4882 scope.go:122] "RemoveContainer" containerID="735e38e17ac0e78df826266f377d2886c71ace06ff3208d568e7e07d511f341d"
Oct 30 22:15:05.360065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.359941    4882 scope.go:122] "RemoveContainer" containerID="43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"
Oct 30 22:15:05.360297 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.360277    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw_odh-ai-gateway-infra(54245ea6-5e26-4b20-b2d8-870dbdb1a180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180"
Oct 30 22:15:05.361892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.361871    4882 scope.go:122] "RemoveContainer" containerID="8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"
Oct 30 22:15:05.362202 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.362178    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz_odh-ai-gateway-infra(1b01b4df-6b9c-4d13-a74d-b9731cf66cec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec"
Oct 30 22:15:05.363430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.363405    4882 generic.go:356] "Generic (PLEG): container finished" podID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerID="01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42" exitCode=60
Oct 30 22:15:05.363523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.363485    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerDied","Data":"01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"}
Oct 30 22:15:05.363719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.363701    4882 scope.go:122] "RemoveContainer" containerID="01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"
Oct 30 22:15:05.364018 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.363995    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp_odh-ai-gateway-infra(ad3c024c-9c76-4caa-9136-44d3eebaf342)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342"
Oct 30 22:15:05.365406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.365384    4882 generic.go:356] "Generic (PLEG): container finished" podID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerID="ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707" exitCode=60
Oct 30 22:15:05.365531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.365464    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerDied","Data":"ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"}
Oct 30 22:15:05.365731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.365711    4882 scope.go:122] "RemoveContainer" containerID="ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"
Oct 30 22:15:05.365997 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.365975    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf_odh-ai-gateway-infra(29fdc1d3-ba80-461d-aa21-2db22628dc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33"
Oct 30 22:15:05.367836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.367624    4882 generic.go:356] "Generic (PLEG): container finished" podID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerID="dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e" exitCode=60
Oct 30 22:15:05.367941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.367899    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerDied","Data":"dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"}
Oct 30 22:15:05.368056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.368041    4882 scope.go:122] "RemoveContainer" containerID="dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"
Oct 30 22:15:05.368325 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.368296    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-knk8n_odh-ai-gateway-infra(2251a9a9-4960-4128-b081-c22ff5f560bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc"
Oct 30 22:15:05.369930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.369910    4882 scope.go:122] "RemoveContainer" containerID="883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"
Oct 30 22:15:05.370231 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.370208    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj_odh-ai-gateway-infra(44e73d51-1357-476b-a4c7-29f3c0a121c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9"
Oct 30 22:15:05.371543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.371518    4882 generic.go:356] "Generic (PLEG): container finished" podID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerID="a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2" exitCode=60
Oct 30 22:15:05.371641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.371615    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerDied","Data":"a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"}
Oct 30 22:15:05.371960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.371921    4882 scope.go:122] "RemoveContainer" containerID="a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"
Oct 30 22:15:05.372189 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.372169    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr_odh-ai-gateway-infra(ac7e81ef-6600-4b3b-870d-8e8d828d15a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1"
Oct 30 22:15:05.374487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.374471    4882 scope.go:122] "RemoveContainer" containerID="9c43b8db994f6aa3ac0dce2bc7143e4d5bd9d6bf0a7b49bcca6c96cf0bc354cf"
Oct 30 22:15:05.376415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.376392    4882 scope.go:122] "RemoveContainer" containerID="86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93"
Oct 30 22:15:05.376683 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.376664    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r_odh-ai-gateway-infra(e2f96be1-7f98-4480-9459-b3411fa98c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c"
Oct 30 22:15:05.377974 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.377954    4882 generic.go:356] "Generic (PLEG): container finished" podID="7e58a043-0b72-4471-8694-b7480513764b" containerID="46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f" exitCode=60
Oct 30 22:15:05.378049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.378032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerDied","Data":"46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"}
Oct 30 22:15:05.378238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.378222    4882 scope.go:122] "RemoveContainer" containerID="46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"
Oct 30 22:15:05.378494 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.378468    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh_odh-ai-gateway-infra(7e58a043-0b72-4471-8694-b7480513764b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" podUID="7e58a043-0b72-4471-8694-b7480513764b"
Oct 30 22:15:05.379992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.379968    4882 generic.go:356] "Generic (PLEG): container finished" podID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerID="234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60" exitCode=60
Oct 30 22:15:05.380110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.380051    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerDied","Data":"234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"}
Oct 30 22:15:05.380364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.380344    4882 scope.go:122] "RemoveContainer" containerID="234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"
Oct 30 22:15:05.380672 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:05.380571    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4_odh-ai-gateway-infra(1c5569be-cfec-43bc-ac28-538fc7de0f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11"
Oct 30 22:15:05.388548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.388525    4882 scope.go:122] "RemoveContainer" containerID="9cdbedd427339887f70ba6fc20ab27a68572093ef77958936871a4e705132421"
Oct 30 22:15:05.402320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.402294    4882 scope.go:122] "RemoveContainer" containerID="5ae34bddd5476b1a2dc1e2af62c35bb91c5b4540d54f48eec1fd2f09e882f6b0"
Oct 30 22:15:05.415814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.415790    4882 scope.go:122] "RemoveContainer" containerID="a5e43ebf0e5c5d7f53e5838668ec4ba2fe2e4389578f072b1b8cc5b6786884e9"
Oct 30 22:15:05.429347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.429328    4882 scope.go:122] "RemoveContainer" containerID="9506826b3d8af61860f49768417c9bd0d3a195e2a25638ed61fc6e63422ef472"
Oct 30 22:15:05.442709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.442691    4882 scope.go:122] "RemoveContainer" containerID="40220b7f8e18e786675bf1cb9188718dd7179ad1466d3ea0ec59fcdcae8cab9e"
Oct 30 22:15:05.747983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:05.747867    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" podStartSLOduration=5.747853211 podStartE2EDuration="5.747853211s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:05.745421954 +0000 UTC m=+2228.918746380" watchObservedRunningTime="2026-10-30 22:15:05.747853211 +0000 UTC m=+2228.921177635"
Oct 30 22:15:06.388862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.388837    4882 scope.go:122] "RemoveContainer" containerID="43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"
Oct 30 22:15:06.389296 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.389143    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw_odh-ai-gateway-infra(54245ea6-5e26-4b20-b2d8-870dbdb1a180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180"
Oct 30 22:15:06.390550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.390526    4882 scope.go:122] "RemoveContainer" containerID="01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"
Oct 30 22:15:06.390906 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.390879    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp_odh-ai-gateway-infra(ad3c024c-9c76-4caa-9136-44d3eebaf342)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342"
Oct 30 22:15:06.392092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.392074    4882 scope.go:122] "RemoveContainer" containerID="ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"
Oct 30 22:15:06.392313 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.392288    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf_odh-ai-gateway-infra(29fdc1d3-ba80-461d-aa21-2db22628dc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33"
Oct 30 22:15:06.393555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.393537    4882 scope.go:122] "RemoveContainer" containerID="dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"
Oct 30 22:15:06.393792 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.393770    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-knk8n_odh-ai-gateway-infra(2251a9a9-4960-4128-b081-c22ff5f560bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc"
Oct 30 22:15:06.395132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.395114    4882 scope.go:122] "RemoveContainer" containerID="a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"
Oct 30 22:15:06.395339 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.395319    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr_odh-ai-gateway-infra(ac7e81ef-6600-4b3b-870d-8e8d828d15a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1"
Oct 30 22:15:06.396692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.396671    4882 scope.go:122] "RemoveContainer" containerID="46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"
Oct 30 22:15:06.396902 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.396883    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh_odh-ai-gateway-infra(7e58a043-0b72-4471-8694-b7480513764b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" podUID="7e58a043-0b72-4471-8694-b7480513764b"
Oct 30 22:15:06.398218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:06.398192    4882 scope.go:122] "RemoveContainer" containerID="234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"
Oct 30 22:15:06.398385 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:06.398369    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4_odh-ai-gateway-infra(1c5569be-cfec-43bc-ac28-538fc7de0f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11"
Oct 30 22:15:07.071710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:07.071678    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"]
Oct 30 22:15:07.076505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:07.076480    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b539382-l6cdd"]
Oct 30 22:15:08.943776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:08.943743    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29463ad4-cbd4-40aa-aa30-147fa738beba" path="/var/lib/kubelet/pods/29463ad4-cbd4-40aa-aa30-147fa738beba/volumes"
Oct 30 22:15:14.944251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:14.944213    4882 scope.go:122] "RemoveContainer" containerID="ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"
Oct 30 22:15:14.944251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:14.944245    4882 scope.go:122] "RemoveContainer" containerID="86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93"
Oct 30 22:15:15.460959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:15.460926    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerStarted","Data":"2f5cc54849b82dabbd0ae648763ddfd1b1ffb968b8a080e8e31338809b2b22c2"}
Oct 30 22:15:15.462966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:15.462940    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerStarted","Data":"746d815b96c0384a665ec5a24051d1146db02bdb1edd7942919361bf8b11ddb8"}
Oct 30 22:15:15.726787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:15.726677    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" podStartSLOduration=15.726658991 podStartE2EDuration="15.726658991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:15:15.722492036 +0000 UTC m=+2238.895816455" watchObservedRunningTime="2026-10-30 22:15:15.726658991 +0000 UTC m=+2238.899983417"
Oct 30 22:15:15.936637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:15.936452    4882 scope.go:122] "RemoveContainer" containerID="46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"
Oct 30 22:15:15.936745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:15.936635    4882 scope.go:122] "RemoveContainer" containerID="8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"
Oct 30 22:15:16.471494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.471449    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerStarted","Data":"f4192ce4e9b519b9ea758731065bffd64cd8677fff204dfadd810c519d7f83c1"}
Oct 30 22:15:16.473466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.473439    4882 generic.go:356] "Generic (PLEG): container finished" podID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerID="2f5cc54849b82dabbd0ae648763ddfd1b1ffb968b8a080e8e31338809b2b22c2" exitCode=60
Oct 30 22:15:16.473633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.473512    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerDied","Data":"2f5cc54849b82dabbd0ae648763ddfd1b1ffb968b8a080e8e31338809b2b22c2"}
Oct 30 22:15:16.473633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.473568    4882 scope.go:122] "RemoveContainer" containerID="ea93a9109db0b5156fb3c204c67ecfdaa786215477222138b2ab86176134a707"
Oct 30 22:15:16.473946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.473928    4882 scope.go:122] "RemoveContainer" containerID="2f5cc54849b82dabbd0ae648763ddfd1b1ffb968b8a080e8e31338809b2b22c2"
Oct 30 22:15:16.474267 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:16.474238    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf_odh-ai-gateway-infra(29fdc1d3-ba80-461d-aa21-2db22628dc33)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33"
Oct 30 22:15:16.475634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.475611    4882 generic.go:356] "Generic (PLEG): container finished" podID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerID="746d815b96c0384a665ec5a24051d1146db02bdb1edd7942919361bf8b11ddb8" exitCode=60
Oct 30 22:15:16.475741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.475630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerDied","Data":"746d815b96c0384a665ec5a24051d1146db02bdb1edd7942919361bf8b11ddb8"}
Oct 30 22:15:16.475959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.475938    4882 scope.go:122] "RemoveContainer" containerID="746d815b96c0384a665ec5a24051d1146db02bdb1edd7942919361bf8b11ddb8"
Oct 30 22:15:16.476207 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:16.476185    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r_odh-ai-gateway-infra(e2f96be1-7f98-4480-9459-b3411fa98c5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c"
Oct 30 22:15:16.477418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.477398    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerStarted","Data":"d3dd109d60143cdfccf90c8fdc98e95f6546e0f17d5a37a99241f9420351e1ae"}
Oct 30 22:15:16.486419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.486396    4882 scope.go:122] "RemoveContainer" containerID="86da5be68e116fae3248461eac572292e9ff60736a91a0f6357b63c79cef0d93"
Oct 30 22:15:16.610049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.609959    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:16.740085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.740048    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:16.936325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.936290    4882 scope.go:122] "RemoveContainer" containerID="883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"
Oct 30 22:15:16.936575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.936408    4882 scope.go:122] "RemoveContainer" containerID="dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"
Oct 30 22:15:16.936575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:16.936487    4882 scope.go:122] "RemoveContainer" containerID="01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"
Oct 30 22:15:17.487904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.487869    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerStarted","Data":"d0d63f8140cc64b119610e756da03aeee1f41c6cfd156224bd9d9e5bac5e324f"}
Oct 30 22:15:17.492745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.492715    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerStarted","Data":"54f27ba28b8a68d144abd93b182444c8391ca367e16cae24457c7a399b5a3f97"}
Oct 30 22:15:17.494843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.494819    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerStarted","Data":"5e27531a41577facfcc535b367a5b5f1901ffb361b57f320124a1126f2e7cd62"}
Oct 30 22:15:17.498532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.498510    4882 generic.go:356] "Generic (PLEG): container finished" podID="7e58a043-0b72-4471-8694-b7480513764b" containerID="d3dd109d60143cdfccf90c8fdc98e95f6546e0f17d5a37a99241f9420351e1ae" exitCode=60
Oct 30 22:15:17.498732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.498713    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerDied","Data":"d3dd109d60143cdfccf90c8fdc98e95f6546e0f17d5a37a99241f9420351e1ae"}
Oct 30 22:15:17.498777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.498760    4882 scope.go:122] "RemoveContainer" containerID="46c57d19552fb4f2407f12e6b5ee2e5f1562b61320049fdd9e8d8f75554b063f"
Oct 30 22:15:17.499092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.499072    4882 scope.go:122] "RemoveContainer" containerID="d3dd109d60143cdfccf90c8fdc98e95f6546e0f17d5a37a99241f9420351e1ae"
Oct 30 22:15:17.499373 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:17.499346    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh_odh-ai-gateway-infra(7e58a043-0b72-4471-8694-b7480513764b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" podUID="7e58a043-0b72-4471-8694-b7480513764b"
Oct 30 22:15:17.501217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.501195    4882 generic.go:356] "Generic (PLEG): container finished" podID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerID="f4192ce4e9b519b9ea758731065bffd64cd8677fff204dfadd810c519d7f83c1" exitCode=60
Oct 30 22:15:17.501313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.501226    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerDied","Data":"f4192ce4e9b519b9ea758731065bffd64cd8677fff204dfadd810c519d7f83c1"}
Oct 30 22:15:17.501504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.501490    4882 scope.go:122] "RemoveContainer" containerID="f4192ce4e9b519b9ea758731065bffd64cd8677fff204dfadd810c519d7f83c1"
Oct 30 22:15:17.501761 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:17.501741    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz_odh-ai-gateway-infra(1b01b4df-6b9c-4d13-a74d-b9731cf66cec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec"
Oct 30 22:15:17.513526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.513501    4882 scope.go:122] "RemoveContainer" containerID="8009bf5eec1dc02946a529a920adece6b25f806457b9292cfcdfd965e1573834"
Oct 30 22:15:17.622490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.622453    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:17.683614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.683557    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:17.716264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.716233    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:17.801902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.801840    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\") pod \"29fdc1d3-ba80-461d-aa21-2db22628dc33\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") "
Oct 30 22:15:17.802088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.802029    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\" (UniqueName: \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\") pod \"e2f96be1-7f98-4480-9459-b3411fa98c5c\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") "
Oct 30 22:15:17.802183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.802169    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\" (UniqueName: \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\") pod \"29fdc1d3-ba80-461d-aa21-2db22628dc33\" (UID: \"29fdc1d3-ba80-461d-aa21-2db22628dc33\") "
Oct 30 22:15:17.802252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.802230    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\") pod \"e2f96be1-7f98-4480-9459-b3411fa98c5c\" (UID: \"e2f96be1-7f98-4480-9459-b3411fa98c5c\") "
Oct 30 22:15:17.802410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.802382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca" pod "29fdc1d3-ba80-461d-aa21-2db22628dc33" (UID: "29fdc1d3-ba80-461d-aa21-2db22628dc33"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:17.802920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.802889    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca" pod "e2f96be1-7f98-4480-9459-b3411fa98c5c" (UID: "e2f96be1-7f98-4480-9459-b3411fa98c5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:17.803103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.803081    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/29fdc1d3-ba80-461d-aa21-2db22628dc33-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.805124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.805095    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5" pod "e2f96be1-7f98-4480-9459-b3411fa98c5c" (UID: "e2f96be1-7f98-4480-9459-b3411fa98c5c"). InnerVolumeSpecName "kube-api-access-zrfc5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:17.805254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.805164    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc" pod "29fdc1d3-ba80-461d-aa21-2db22628dc33" (UID: "29fdc1d3-ba80-461d-aa21-2db22628dc33"). InnerVolumeSpecName "kube-api-access-mgqvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:17.904036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.903944    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zrfc5\" (UniqueName: \"kubernetes.io/projected/e2f96be1-7f98-4480-9459-b3411fa98c5c-kube-api-access-zrfc5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.904036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.903975    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgqvc\" (UniqueName: \"kubernetes.io/projected/29fdc1d3-ba80-461d-aa21-2db22628dc33-kube-api-access-mgqvc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.904036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.903986    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2f96be1-7f98-4480-9459-b3411fa98c5c-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:17.936007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.935979    4882 scope.go:122] "RemoveContainer" containerID="43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"
Oct 30 22:15:17.936204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.936175    4882 scope.go:122] "RemoveContainer" containerID="234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"
Oct 30 22:15:17.936351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.936318    4882 scope.go:122] "RemoveContainer" containerID="a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"
Oct 30 22:15:17.975390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:17.975358    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:18.510862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.510759    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerStarted","Data":"d9794f37fb340c91d2b205ea85d8b261f2f9c4d51460b95b7d6cda4fa4105ca9"}
Oct 30 22:15:18.515839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.515810    4882 generic.go:356] "Generic (PLEG): container finished" podID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerID="d0d63f8140cc64b119610e756da03aeee1f41c6cfd156224bd9d9e5bac5e324f" exitCode=60
Oct 30 22:15:18.515969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.515884    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerDied","Data":"d0d63f8140cc64b119610e756da03aeee1f41c6cfd156224bd9d9e5bac5e324f"}
Oct 30 22:15:18.515969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.515953    4882 scope.go:122] "RemoveContainer" containerID="01f4fd8a9603afecbeb3da4f52e36a461997b4463ba2a591fa7e53c321d85f42"
Oct 30 22:15:18.516326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.516307    4882 scope.go:122] "RemoveContainer" containerID="d0d63f8140cc64b119610e756da03aeee1f41c6cfd156224bd9d9e5bac5e324f"
Oct 30 22:15:18.516668 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:18.516638    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp_odh-ai-gateway-infra(ad3c024c-9c76-4caa-9136-44d3eebaf342)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342"
Oct 30 22:15:18.517996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.517979    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"
Oct 30 22:15:18.518123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.518018    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf" event={"ID":"29fdc1d3-ba80-461d-aa21-2db22628dc33","Type":"ContainerDied","Data":"427a119446d8bdd557400f73bdab84846372f193d6d5946dd936905ca53f5724"}
Oct 30 22:15:18.520130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.520107    4882 generic.go:356] "Generic (PLEG): container finished" podID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerID="54f27ba28b8a68d144abd93b182444c8391ca367e16cae24457c7a399b5a3f97" exitCode=60
Oct 30 22:15:18.520242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.520190    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerDied","Data":"54f27ba28b8a68d144abd93b182444c8391ca367e16cae24457c7a399b5a3f97"}
Oct 30 22:15:18.520597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.520558    4882 scope.go:122] "RemoveContainer" containerID="54f27ba28b8a68d144abd93b182444c8391ca367e16cae24457c7a399b5a3f97"
Oct 30 22:15:18.520893 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:18.520868    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889975-knk8n_odh-ai-gateway-infra(2251a9a9-4960-4128-b081-c22ff5f560bc)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc"
Oct 30 22:15:18.522539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.522514    4882 generic.go:356] "Generic (PLEG): container finished" podID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerID="5e27531a41577facfcc535b367a5b5f1901ffb361b57f320124a1126f2e7cd62" exitCode=60
Oct 30 22:15:18.522675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.522646    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerDied","Data":"5e27531a41577facfcc535b367a5b5f1901ffb361b57f320124a1126f2e7cd62"}
Oct 30 22:15:18.523099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.523070    4882 scope.go:122] "RemoveContainer" containerID="5e27531a41577facfcc535b367a5b5f1901ffb361b57f320124a1126f2e7cd62"
Oct 30 22:15:18.523374 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:18.523351    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj_odh-ai-gateway-infra(44e73d51-1357-476b-a4c7-29f3c0a121c9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9"
Oct 30 22:15:18.524582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.524291    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"
Oct 30 22:15:18.524582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.524299    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r" event={"ID":"e2f96be1-7f98-4480-9459-b3411fa98c5c","Type":"ContainerDied","Data":"8817ce8fc31445b1756b3c3fae1e4517f7e14af97afdf4af182ad96010e84985"}
Oct 30 22:15:18.532769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.532733    4882 scope.go:122] "RemoveContainer" containerID="2f5cc54849b82dabbd0ae648763ddfd1b1ffb968b8a080e8e31338809b2b22c2"
Oct 30 22:15:18.544770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.544747    4882 scope.go:122] "RemoveContainer" containerID="dfcf297efe35c31b73f8465871f827801df492afd589614322b58e9e6184f18e"
Oct 30 22:15:18.559733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.559711    4882 scope.go:122] "RemoveContainer" containerID="883cf12a611d5b2ad5ac92c14fd868f8bda981b4c261ea48343c7dcdfcc960a5"
Oct 30 22:15:18.574755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.574723    4882 scope.go:122] "RemoveContainer" containerID="746d815b96c0384a665ec5a24051d1146db02bdb1edd7942919361bf8b11ddb8"
Oct 30 22:15:18.621771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.619999    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:18.708776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.708744    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:18.745914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.745876    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:18.752244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.752218    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:18.818646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.818548    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\" (UniqueName: \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\") pod \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") "
Oct 30 22:15:18.818793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.818697    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\") pod \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\" (UID: \"1b01b4df-6b9c-4d13-a74d-b9731cf66cec\") "
Oct 30 22:15:18.819069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.819040    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca" pod "1b01b4df-6b9c-4d13-a74d-b9731cf66cec" (UID: "1b01b4df-6b9c-4d13-a74d-b9731cf66cec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.819354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.819335    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.820845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.820823    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8" pod "1b01b4df-6b9c-4d13-a74d-b9731cf66cec" (UID: "1b01b4df-6b9c-4d13-a74d-b9731cf66cec"). InnerVolumeSpecName "kube-api-access-4cnb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.921039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.921009    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\") pod \"7e58a043-0b72-4471-8694-b7480513764b\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") "
Oct 30 22:15:18.921224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.921155    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\" (UniqueName: \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\") pod \"7e58a043-0b72-4471-8694-b7480513764b\" (UID: \"7e58a043-0b72-4471-8694-b7480513764b\") "
Oct 30 22:15:18.921453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.921417    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca" pod "7e58a043-0b72-4471-8694-b7480513764b" (UID: "7e58a043-0b72-4471-8694-b7480513764b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:18.921699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.921678    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e58a043-0b72-4471-8694-b7480513764b-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.921799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.921700    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnb8\" (UniqueName: \"kubernetes.io/projected/1b01b4df-6b9c-4d13-a74d-b9731cf66cec-kube-api-access-4cnb8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:18.923646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.923623    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng" pod "7e58a043-0b72-4471-8694-b7480513764b" (UID: "7e58a043-0b72-4471-8694-b7480513764b"). InnerVolumeSpecName "kube-api-access-nkbng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:18.981398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.981349    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:18.984654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:18.984631    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-a4257d-29889975-rz4hf"]
Oct 30 22:15:19.022858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.022825    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nkbng\" (UniqueName: \"kubernetes.io/projected/7e58a043-0b72-4471-8694-b7480513764b-kube-api-access-nkbng\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:19.105663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.105628    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:19.115916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.115879    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-5d2b13-29889975-2bp2r"]
Oct 30 22:15:19.129429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.129402    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:19.538633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.538512    4882 generic.go:356] "Generic (PLEG): container finished" podID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerID="d9794f37fb340c91d2b205ea85d8b261f2f9c4d51460b95b7d6cda4fa4105ca9" exitCode=60
Oct 30 22:15:19.538633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.538617    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerDied","Data":"d9794f37fb340c91d2b205ea85d8b261f2f9c4d51460b95b7d6cda4fa4105ca9"}
Oct 30 22:15:19.539169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.538694    4882 scope.go:122] "RemoveContainer" containerID="43bf2586ebcb601dca8d638d6203da7fe7fabe2af9d025385f5fb0ae078925f6"
Oct 30 22:15:19.539169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.539081    4882 scope.go:122] "RemoveContainer" containerID="d9794f37fb340c91d2b205ea85d8b261f2f9c4d51460b95b7d6cda4fa4105ca9"
Oct 30 22:15:19.540666 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:19.539422    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw_odh-ai-gateway-infra(54245ea6-5e26-4b20-b2d8-870dbdb1a180)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180"
Oct 30 22:15:19.540666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.540543    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz" event={"ID":"1b01b4df-6b9c-4d13-a74d-b9731cf66cec","Type":"ContainerDied","Data":"28ff8013e90f7ef9476a793d616a66cf6f27ec60fbb47c7d3d8106033c8a83c9"}
Oct 30 22:15:19.541245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.540678    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"
Oct 30 22:15:19.564240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.555930    4882 scope.go:122] "RemoveContainer" containerID="f4192ce4e9b519b9ea758731065bffd64cd8677fff204dfadd810c519d7f83c1"
Oct 30 22:15:19.568266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.565006    4882 generic.go:356] "Generic (PLEG): container finished" podID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerID="4c7fd54dd39ef95c277d7d898251bfc37fb5353afdd47bb9f82f88b9259898a7" exitCode=60
Oct 30 22:15:19.568266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.565128    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerDied","Data":"4c7fd54dd39ef95c277d7d898251bfc37fb5353afdd47bb9f82f88b9259898a7"}
Oct 30 22:15:19.568266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.565621    4882 scope.go:122] "RemoveContainer" containerID="4c7fd54dd39ef95c277d7d898251bfc37fb5353afdd47bb9f82f88b9259898a7"
Oct 30 22:15:19.568266 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:19.566076    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr_odh-ai-gateway-infra(ac7e81ef-6600-4b3b-870d-8e8d828d15a1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1"
Oct 30 22:15:19.571519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.571482    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh" event={"ID":"7e58a043-0b72-4471-8694-b7480513764b","Type":"ContainerDied","Data":"d07f389dd46996257f4b328366510979e870253b071b082c4a11cefe2d454da3"}
Oct 30 22:15:19.571641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.571622    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"
Oct 30 22:15:19.576652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.576628    4882 scope.go:122] "RemoveContainer" containerID="a11b3a55b3c0de09cadf64dd085a9e799e079424f47227c5b38311e0b11a7ff2"
Oct 30 22:15:19.577113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.577091    4882 generic.go:356] "Generic (PLEG): container finished" podID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerID="efeeb87cd0b3deacace22f81fea0c78e3bfe9687feab7c2947b42196d69390e4" exitCode=60
Oct 30 22:15:19.577264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.577125    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerDied","Data":"efeeb87cd0b3deacace22f81fea0c78e3bfe9687feab7c2947b42196d69390e4"}
Oct 30 22:15:19.577474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.577457    4882 scope.go:122] "RemoveContainer" containerID="efeeb87cd0b3deacace22f81fea0c78e3bfe9687feab7c2947b42196d69390e4"
Oct 30 22:15:19.578186 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:19.578163    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4_odh-ai-gateway-infra(1c5569be-cfec-43bc-ac28-538fc7de0f11)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11"
Oct 30 22:15:19.613301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.613271    4882 scope.go:122] "RemoveContainer" containerID="d3dd109d60143cdfccf90c8fdc98e95f6546e0f17d5a37a99241f9420351e1ae"
Oct 30 22:15:19.637645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.637092    4882 scope.go:122] "RemoveContainer" containerID="234cbc6f04be83a18885a4101aabef7d5547c1e861c983ad32fafee37610cc60"
Oct 30 22:15:19.787367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.787336    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:19.808497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.808424    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:19.826517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.826485    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:19.938242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938212    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\" (UniqueName: \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\") pod \"2251a9a9-4960-4128-b081-c22ff5f560bc\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") "
Oct 30 22:15:19.938432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938283    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\" (UniqueName: \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\") pod \"ad3c024c-9c76-4caa-9136-44d3eebaf342\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") "
Oct 30 22:15:19.938432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938329    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\") pod \"44e73d51-1357-476b-a4c7-29f3c0a121c9\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") "
Oct 30 22:15:19.938432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938406    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\") pod \"2251a9a9-4960-4128-b081-c22ff5f560bc\" (UID: \"2251a9a9-4960-4128-b081-c22ff5f560bc\") "
Oct 30 22:15:19.938776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938447    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\" (UniqueName: \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\") pod \"44e73d51-1357-476b-a4c7-29f3c0a121c9\" (UID: \"44e73d51-1357-476b-a4c7-29f3c0a121c9\") "
Oct 30 22:15:19.938905 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.938836    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\") pod \"ad3c024c-9c76-4caa-9136-44d3eebaf342\" (UID: \"ad3c024c-9c76-4caa-9136-44d3eebaf342\") "
Oct 30 22:15:19.939552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.939524    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca" pod "ad3c024c-9c76-4caa-9136-44d3eebaf342" (UID: "ad3c024c-9c76-4caa-9136-44d3eebaf342"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.942091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.942065    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca" pod "2251a9a9-4960-4128-b081-c22ff5f560bc" (UID: "2251a9a9-4960-4128-b081-c22ff5f560bc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.942258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.942068    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca" pod "44e73d51-1357-476b-a4c7-29f3c0a121c9" (UID: "44e73d51-1357-476b-a4c7-29f3c0a121c9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:19.942258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.942112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b" pod "2251a9a9-4960-4128-b081-c22ff5f560bc" (UID: "2251a9a9-4960-4128-b081-c22ff5f560bc"). InnerVolumeSpecName "kube-api-access-r9x6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.945066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.945033    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf" pod "ad3c024c-9c76-4caa-9136-44d3eebaf342" (UID: "ad3c024c-9c76-4caa-9136-44d3eebaf342"). InnerVolumeSpecName "kube-api-access-rwnrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:19.945173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:19.945073    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4" pod "44e73d51-1357-476b-a4c7-29f3c0a121c9" (UID: "44e73d51-1357-476b-a4c7-29f3c0a121c9"). InnerVolumeSpecName "kube-api-access-llzt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.002304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.002271    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:20.006205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.006180    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-c4bfac-29889975-blxqh"]
Oct 30 22:15:20.040373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040340    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9x6b\" (UniqueName: \"kubernetes.io/projected/2251a9a9-4960-4128-b081-c22ff5f560bc-kube-api-access-r9x6b\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.040373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040368    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwnrf\" (UniqueName: \"kubernetes.io/projected/ad3c024c-9c76-4caa-9136-44d3eebaf342-kube-api-access-rwnrf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.040373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040381    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44e73d51-1357-476b-a4c7-29f3c0a121c9-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.040577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040390    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2251a9a9-4960-4128-b081-c22ff5f560bc-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.040577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040399    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llzt4\" (UniqueName: \"kubernetes.io/projected/44e73d51-1357-476b-a4c7-29f3c0a121c9-kube-api-access-llzt4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.040577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.040408    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ad3c024c-9c76-4caa-9136-44d3eebaf342-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.193263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.193228    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:20.197305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.197280    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889975-bw6mz"]
Oct 30 22:15:20.278221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.278180    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:20.589430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.589401    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"
Oct 30 22:15:20.589963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.589401    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp" event={"ID":"ad3c024c-9c76-4caa-9136-44d3eebaf342","Type":"ContainerDied","Data":"47ba3ac98dbb73bf205b587e4f00db9e4de0ac17d268d7647af773e19985ec71"}
Oct 30 22:15:20.589963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.589532    4882 scope.go:122] "RemoveContainer" containerID="d0d63f8140cc64b119610e756da03aeee1f41c6cfd156224bd9d9e5bac5e324f"
Oct 30 22:15:20.590816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.590797    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"
Oct 30 22:15:20.590902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.590847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n" event={"ID":"2251a9a9-4960-4128-b081-c22ff5f560bc","Type":"ContainerDied","Data":"e6d60ed076e8cb76e37a9f3900570135806e0a9423379bbaf8c20a3f54541963"}
Oct 30 22:15:20.592673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.592644    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj" event={"ID":"44e73d51-1357-476b-a4c7-29f3c0a121c9","Type":"ContainerDied","Data":"6bacdd33b2fc63d4365b9781287ffc93684516d1da70436f6e070894c113806d"}
Oct 30 22:15:20.593021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.592999    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"
Oct 30 22:15:20.602066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.601924    4882 scope.go:122] "RemoveContainer" containerID="54f27ba28b8a68d144abd93b182444c8391ca367e16cae24457c7a399b5a3f97"
Oct 30 22:15:20.613907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.613886    4882 scope.go:122] "RemoveContainer" containerID="5e27531a41577facfcc535b367a5b5f1901ffb361b57f320124a1126f2e7cd62"
Oct 30 22:15:20.765405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.765374    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:20.809740 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:20.809700    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:15:20.828826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.828795    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:20.947211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.947173    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" path="/var/lib/kubelet/pods/1b01b4df-6b9c-4d13-a74d-b9731cf66cec/volumes"
Oct 30 22:15:20.947878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.947856    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" path="/var/lib/kubelet/pods/29fdc1d3-ba80-461d-aa21-2db22628dc33/volumes"
Oct 30 22:15:20.948436 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.948416    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e58a043-0b72-4471-8694-b7480513764b" path="/var/lib/kubelet/pods/7e58a043-0b72-4471-8694-b7480513764b/volumes"
Oct 30 22:15:20.949480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.949458    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" path="/var/lib/kubelet/pods/e2f96be1-7f98-4480-9459-b3411fa98c5c/volumes"
Oct 30 22:15:20.951609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.951562    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\" (UniqueName: \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\") pod \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") "
Oct 30 22:15:20.951780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.951751    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\") pod \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\" (UID: \"54245ea6-5e26-4b20-b2d8-870dbdb1a180\") "
Oct 30 22:15:20.952246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.952222    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca" pod "54245ea6-5e26-4b20-b2d8-870dbdb1a180" (UID: "54245ea6-5e26-4b20-b2d8-870dbdb1a180"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:20.953109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.953065    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/54245ea6-5e26-4b20-b2d8-870dbdb1a180-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:20.954211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.954189    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8" pod "54245ea6-5e26-4b20-b2d8-870dbdb1a180" (UID: "54245ea6-5e26-4b20-b2d8-870dbdb1a180"). InnerVolumeSpecName "kube-api-access-ngvt8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:20.988824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.988797    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:20.990356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.990331    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:20.992698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:20.992673    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889975-knk8n"]
Oct 30 22:15:21.053992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.053952    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngvt8\" (UniqueName: \"kubernetes.io/projected/54245ea6-5e26-4b20-b2d8-870dbdb1a180-kube-api-access-ngvt8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.155113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.155033    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\") pod \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") "
Oct 30 22:15:21.155325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.155160    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\" (UniqueName: \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\") pod \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\" (UID: \"ac7e81ef-6600-4b3b-870d-8e8d828d15a1\") "
Oct 30 22:15:21.155460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.155438    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca" pod "ac7e81ef-6600-4b3b-870d-8e8d828d15a1" (UID: "ac7e81ef-6600-4b3b-870d-8e8d828d15a1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.155687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.155671    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.157274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.157252    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd" pod "ac7e81ef-6600-4b3b-870d-8e8d828d15a1" (UID: "ac7e81ef-6600-4b3b-870d-8e8d828d15a1"). InnerVolumeSpecName "kube-api-access-skjwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.167413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.167382    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:21.171925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.171902    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-b-138566-29889975-s74nj"]
Oct 30 22:15:21.256935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.256901    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skjwd\" (UniqueName: \"kubernetes.io/projected/ac7e81ef-6600-4b3b-870d-8e8d828d15a1-kube-api-access-skjwd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.367642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.367602    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:21.374602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.374553    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-ipp-a-65659d-29889975-rcvjp"]
Oct 30 22:15:21.585909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.585872    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:21.610173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.610141    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"
Oct 30 22:15:21.611419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.610189    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr" event={"ID":"ac7e81ef-6600-4b3b-870d-8e8d828d15a1","Type":"ContainerDied","Data":"b4d5321b98518e9dab655ce6da788f7c3ae3c1e3bfea06f385eae2830c08ecf4"}
Oct 30 22:15:21.611578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.611473    4882 scope.go:122] "RemoveContainer" containerID="4c7fd54dd39ef95c277d7d898251bfc37fb5353afdd47bb9f82f88b9259898a7"
Oct 30 22:15:21.612420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.612388    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw" event={"ID":"54245ea6-5e26-4b20-b2d8-870dbdb1a180","Type":"ContainerDied","Data":"c01a8fc46f542c5a6fa706c8ba0ee2ceea8192a871eb1e838c3d12ebd200834e"}
Oct 30 22:15:21.612739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.612525    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"
Oct 30 22:15:21.629218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.629198    4882 scope.go:122] "RemoveContainer" containerID="d9794f37fb340c91d2b205ea85d8b261f2f9c4d51460b95b7d6cda4fa4105ca9"
Oct 30 22:15:21.766915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.766885    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:21.864771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.864685    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\") pod \"1c5569be-cfec-43bc-ac28-538fc7de0f11\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") "
Oct 30 22:15:21.864771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.864765    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\" (UniqueName: \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\") pod \"1c5569be-cfec-43bc-ac28-538fc7de0f11\" (UID: \"1c5569be-cfec-43bc-ac28-538fc7de0f11\") "
Oct 30 22:15:21.865129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.865104    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca" pod "1c5569be-cfec-43bc-ac28-538fc7de0f11" (UID: "1c5569be-cfec-43bc-ac28-538fc7de0f11"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:21.865317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.865302    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1c5569be-cfec-43bc-ac28-538fc7de0f11-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:21.866987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.866957    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs" pod "1c5569be-cfec-43bc-ac28-538fc7de0f11" (UID: "1c5569be-cfec-43bc-ac28-538fc7de0f11"). InnerVolumeSpecName "kube-api-access-9hljs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:21.885138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.885109    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:21.889110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.889081    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-a3aa65-29889975-6v9zw"]
Oct 30 22:15:21.966308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:21.966273    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9hljs\" (UniqueName: \"kubernetes.io/projected/1c5569be-cfec-43bc-ac28-538fc7de0f11-kube-api-access-9hljs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:22.001209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.001177    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:22.006733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.006705    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889975-jhmxr"]
Oct 30 22:15:22.625542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.625511    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"
Oct 30 22:15:22.625542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.625532    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4" event={"ID":"1c5569be-cfec-43bc-ac28-538fc7de0f11","Type":"ContainerDied","Data":"94f526631376e6c52c922fb5155723eb221aafc3af696cf67e269a9f6d47731d"}
Oct 30 22:15:22.626142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.625611    4882 scope.go:122] "RemoveContainer" containerID="efeeb87cd0b3deacace22f81fea0c78e3bfe9687feab7c2947b42196d69390e4"
Oct 30 22:15:22.801697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.801665    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:22.807311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.807279    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889975-j2nh4"]
Oct 30 22:15:22.943852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.943777    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" path="/var/lib/kubelet/pods/1c5569be-cfec-43bc-ac28-538fc7de0f11/volumes"
Oct 30 22:15:22.944218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.944205    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" path="/var/lib/kubelet/pods/2251a9a9-4960-4128-b081-c22ff5f560bc/volumes"
Oct 30 22:15:22.944598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.944575    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" path="/var/lib/kubelet/pods/44e73d51-1357-476b-a4c7-29f3c0a121c9/volumes"
Oct 30 22:15:22.945272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.945256    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" path="/var/lib/kubelet/pods/54245ea6-5e26-4b20-b2d8-870dbdb1a180/volumes"
Oct 30 22:15:22.945673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.945659    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" path="/var/lib/kubelet/pods/ac7e81ef-6600-4b3b-870d-8e8d828d15a1/volumes"
Oct 30 22:15:22.946040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:22.946028    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" path="/var/lib/kubelet/pods/ad3c024c-9c76-4caa-9136-44d3eebaf342/volumes"
Oct 30 22:15:27.200413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:27.200376    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:15:49.135185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:49.135104    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:15:49.244069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:49.244037    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:15:50.808992 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:50.808957    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:15:54.136010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.135948    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" podUID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" containerName="payload-processing" containerID="cri-o://bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375" gracePeriod=25
Oct 30 22:15:54.244905 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.244853    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" podUID="3a02045c-9b27-4263-a592-1dc5793f6123" containerName="payload-pre-processing" containerID="cri-o://fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37" gracePeriod=25
Oct 30 22:15:54.416555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.416526    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:15:54.448328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.448293    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\" (UniqueName: \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\") pod \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") "
Oct 30 22:15:54.448506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.448343    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\") pod \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\" (UID: \"2ba52d04-6f37-4a45-90ca-0e85ea2a660c\") "
Oct 30 22:15:54.448830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.448803    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume" pod "2ba52d04-6f37-4a45-90ca-0e85ea2a660c" (UID: "2ba52d04-6f37-4a45-90ca-0e85ea2a660c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:54.450665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.450625    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9" pod "2ba52d04-6f37-4a45-90ca-0e85ea2a660c" (UID: "2ba52d04-6f37-4a45-90ca-0e85ea2a660c"). InnerVolumeSpecName "kube-api-access-5xgz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:54.513546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.513518    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:15:54.550156 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550111    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\" (UniqueName: \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\") pod \"3a02045c-9b27-4263-a592-1dc5793f6123\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") "
Oct 30 22:15:54.550345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550216    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\") pod \"3a02045c-9b27-4263-a592-1dc5793f6123\" (UID: \"3a02045c-9b27-4263-a592-1dc5793f6123\") "
Oct 30 22:15:54.550699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550667    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume" pod "3a02045c-9b27-4263-a592-1dc5793f6123" (UID: "3a02045c-9b27-4263-a592-1dc5793f6123"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:15:54.550834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550790    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xgz9\" (UniqueName: \"kubernetes.io/projected/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-kube-api-access-5xgz9\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.550834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550810    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ba52d04-6f37-4a45-90ca-0e85ea2a660c-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.550834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.550825    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3a02045c-9b27-4263-a592-1dc5793f6123-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.552349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.552317    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd" pod "3a02045c-9b27-4263-a592-1dc5793f6123" (UID: "3a02045c-9b27-4263-a592-1dc5793f6123"). InnerVolumeSpecName "kube-api-access-24rcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:15:54.652174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.652094    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24rcd\" (UniqueName: \"kubernetes.io/projected/3a02045c-9b27-4263-a592-1dc5793f6123-kube-api-access-24rcd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:15:54.850509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.850472    4882 generic.go:356] "Generic (PLEG): container finished" podID="3a02045c-9b27-4263-a592-1dc5793f6123" containerID="fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37" exitCode=0
Oct 30 22:15:54.850736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.850529    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"
Oct 30 22:15:54.850736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.850561    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" event={"ID":"3a02045c-9b27-4263-a592-1dc5793f6123","Type":"ContainerDied","Data":"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"}
Oct 30 22:15:54.850736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.850635    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm" event={"ID":"3a02045c-9b27-4263-a592-1dc5793f6123","Type":"ContainerDied","Data":"f235912972218a0f1489f12935a47aa742281f1f2b409514437878e254d00dcd"}
Oct 30 22:15:54.850736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.850658    4882 scope.go:122] "RemoveContainer" containerID="fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"
Oct 30 22:15:54.852026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.852004    4882 generic.go:356] "Generic (PLEG): container finished" podID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" containerID="bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375" exitCode=0
Oct 30 22:15:54.852143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.852046    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"
Oct 30 22:15:54.852143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.852089    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" event={"ID":"2ba52d04-6f37-4a45-90ca-0e85ea2a660c","Type":"ContainerDied","Data":"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"}
Oct 30 22:15:54.852143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.852136    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7" event={"ID":"2ba52d04-6f37-4a45-90ca-0e85ea2a660c","Type":"ContainerDied","Data":"5d41b1e5c85902e6a597b3f2d77d3e6c0e837f84a8cdcccefe9386adae5142a4"}
Oct 30 22:15:54.862505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.862468    4882 scope.go:122] "RemoveContainer" containerID="fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"
Oct 30 22:15:54.862893 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:54.862869    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37\": container with ID starting with fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37 not found: ID does not exist" containerID="fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"
Oct 30 22:15:54.862988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.862901    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37"} err="failed to get container status \"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37\": rpc error: code = NotFound desc = could not find container \"fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37\": container with ID starting with fc4e3000dec00a1a1bb70c5bac865be6ef593b758c3941793ed4c7cabe483f37 not found: ID does not exist"
Oct 30 22:15:54.862988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.862918    4882 scope.go:122] "RemoveContainer" containerID="bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"
Oct 30 22:15:54.872685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.872663    4882 scope.go:122] "RemoveContainer" containerID="bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"
Oct 30 22:15:54.873007 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:15:54.872986    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375\": container with ID starting with bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375 not found: ID does not exist" containerID="bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"
Oct 30 22:15:54.873064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:54.873017    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375"} err="failed to get container status \"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375\": rpc error: code = NotFound desc = could not find container \"bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375\": container with ID starting with bb81d3fd8a4f80b72a4b300793e6e7f2844297682657f6fa1a7f7c03b5191375 not found: ID does not exist"
Oct 30 22:15:55.018369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:55.018284    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:15:55.022639 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:55.022577    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-5fbd49cb5f-7d7pm"]
Oct 30 22:15:55.133290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:55.133249    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:15:55.142491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:55.142454    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-d858f6565-wfcd7"]
Oct 30 22:15:56.943164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:56.943133    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" path="/var/lib/kubelet/pods/2ba52d04-6f37-4a45-90ca-0e85ea2a660c/volumes"
Oct 30 22:15:56.943563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:56.943548    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a02045c-9b27-4263-a592-1dc5793f6123" path="/var/lib/kubelet/pods/3a02045c-9b27-4263-a592-1dc5793f6123/volumes"
Oct 30 22:15:58.465791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:58.465756    4882 scope.go:122] "RemoveContainer" containerID="fb201f4365bfd362e58322167227ad6b43536d836d3b643d20cafa125759e22a"
Oct 30 22:15:58.476162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:15:58.476142    4882 scope.go:122] "RemoveContainer" containerID="b485f9b600503fa665ab13c95507f7c30a251670e0b5b89cc2ebd65951c3f097"
Oct 30 22:16:05.055110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:05.055077    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:16:07.213464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:07.213430    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:16:07.320062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:07.320027    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:16:12.214549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.214504    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" podUID="0eced86d-2e75-48f9-a98f-218e397abc97" containerName="payload-processing" containerID="cri-o://4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d" gracePeriod=25
Oct 30 22:16:12.320365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.320306    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" podUID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" containerName="payload-pre-processing" containerID="cri-o://901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371" gracePeriod=25
Oct 30 22:16:12.510612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.510547    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:16:12.611780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.611752    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:16:12.668980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.668944    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\" (UniqueName: \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\") pod \"0eced86d-2e75-48f9-a98f-218e397abc97\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") "
Oct 30 22:16:12.669150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.669029    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\") pod \"0eced86d-2e75-48f9-a98f-218e397abc97\" (UID: \"0eced86d-2e75-48f9-a98f-218e397abc97\") "
Oct 30 22:16:12.669434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.669403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume" pod "0eced86d-2e75-48f9-a98f-218e397abc97" (UID: "0eced86d-2e75-48f9-a98f-218e397abc97"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:12.671058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.671036    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8" pod "0eced86d-2e75-48f9-a98f-218e397abc97" (UID: "0eced86d-2e75-48f9-a98f-218e397abc97"). InnerVolumeSpecName "kube-api-access-z95b8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:12.769912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.769825    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\") pod \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") "
Oct 30 22:16:12.770100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.770007    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\" (UniqueName: \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\") pod \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\" (UID: \"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a\") "
Oct 30 22:16:12.770265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.770240    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume" pod "47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" (UID: "47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:12.770446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.770428    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.770502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.770452    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z95b8\" (UniqueName: \"kubernetes.io/projected/0eced86d-2e75-48f9-a98f-218e397abc97-kube-api-access-z95b8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.770502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.770467    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0eced86d-2e75-48f9-a98f-218e397abc97-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.772194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.772170    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6" pod "47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" (UID: "47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a"). InnerVolumeSpecName "kube-api-access-9c7q6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:12.871805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.871765    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7q6\" (UniqueName: \"kubernetes.io/projected/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a-kube-api-access-9c7q6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:12.987370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.987330    4882 generic.go:356] "Generic (PLEG): container finished" podID="0eced86d-2e75-48f9-a98f-218e397abc97" containerID="4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d" exitCode=0
Oct 30 22:16:12.987370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.987373    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"
Oct 30 22:16:12.987570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.987413    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" event={"ID":"0eced86d-2e75-48f9-a98f-218e397abc97","Type":"ContainerDied","Data":"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"}
Oct 30 22:16:12.987570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.987467    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2" event={"ID":"0eced86d-2e75-48f9-a98f-218e397abc97","Type":"ContainerDied","Data":"91e7b765e7227890e558ac06da549ebf587787ff01828bdbd5b9f6494a2db29b"}
Oct 30 22:16:12.987570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.987498    4882 scope.go:122] "RemoveContainer" containerID="4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"
Oct 30 22:16:12.989066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.989041    4882 generic.go:356] "Generic (PLEG): container finished" podID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" containerID="901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371" exitCode=0
Oct 30 22:16:12.989183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.989084    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"
Oct 30 22:16:12.989183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.989103    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" event={"ID":"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a","Type":"ContainerDied","Data":"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"}
Oct 30 22:16:12.989183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.989145    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q" event={"ID":"47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a","Type":"ContainerDied","Data":"656817cdb24d52591f44ef8d53a259f9f3a1b1cf918c2a3b4a619497a7694650"}
Oct 30 22:16:12.997697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.997576    4882 scope.go:122] "RemoveContainer" containerID="4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"
Oct 30 22:16:12.997984 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:12.997965    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d\": container with ID starting with 4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d not found: ID does not exist" containerID="4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"
Oct 30 22:16:12.998035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.997996    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d"} err="failed to get container status \"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d\": rpc error: code = NotFound desc = could not find container \"4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d\": container with ID starting with 4b00482837661cc1ed432efefdd35aae1eaba89dee559e19220bc4357d1dd15d not found: ID does not exist"
Oct 30 22:16:12.998035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:12.998012    4882 scope.go:122] "RemoveContainer" containerID="901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"
Oct 30 22:16:13.006839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.006820    4882 scope.go:122] "RemoveContainer" containerID="901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"
Oct 30 22:16:13.007144 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:13.007123    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371\": container with ID starting with 901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371 not found: ID does not exist" containerID="901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"
Oct 30 22:16:13.007219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.007151    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371"} err="failed to get container status \"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371\": rpc error: code = NotFound desc = could not find container \"901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371\": container with ID starting with 901ba478ac7476829e8f801338723e23d0b650684bf3878821bb54bbcc990371 not found: ID does not exist"
Oct 30 22:16:13.156080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.156041    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:16:13.161303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.161271    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-5c85fcfd85-mqgq2"]
Oct 30 22:16:13.275025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.274990    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:16:13.278321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:13.278295    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-57b779467f-tkc7q"]
Oct 30 22:16:14.943670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:14.943640    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0eced86d-2e75-48f9-a98f-218e397abc97" path="/var/lib/kubelet/pods/0eced86d-2e75-48f9-a98f-218e397abc97/volumes"
Oct 30 22:16:14.944084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:14.944068    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" path="/var/lib/kubelet/pods/47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a/volumes"
Oct 30 22:16:20.808557 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:20.808517    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:16:21.735765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:21.735723    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:16:21.843382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:21.843349    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:16:26.736675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:26.736625    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" podUID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" containerName="payload-processing" containerID="cri-o://9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802" gracePeriod=25
Oct 30 22:16:26.844649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:26.844567    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" podUID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" containerName="payload-pre-processing" containerID="cri-o://8642a72afdb6d080255d24346944885535d6c12b51ca0625bbffb623e5c21736" gracePeriod=25
Oct 30 22:16:27.007839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.007767    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:16:27.046514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.037608    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\" (UniqueName: \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\") pod \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") "
Oct 30 22:16:27.046514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.037686    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\") pod \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\" (UID: \"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd\") "
Oct 30 22:16:27.079503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.066636    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume" pod "713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" (UID: "713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:27.079503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.067007    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h" pod "713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" (UID: "713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd"). InnerVolumeSpecName "kube-api-access-r6g9h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:27.091287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.091245    4882 generic.go:356] "Generic (PLEG): container finished" podID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" containerID="8642a72afdb6d080255d24346944885535d6c12b51ca0625bbffb623e5c21736" exitCode=0
Oct 30 22:16:27.091574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.091318    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" event={"ID":"dbc916b9-d00a-495a-993d-2c77bb1cd2a3","Type":"ContainerDied","Data":"8642a72afdb6d080255d24346944885535d6c12b51ca0625bbffb623e5c21736"}
Oct 30 22:16:27.093553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.093521    4882 generic.go:356] "Generic (PLEG): container finished" podID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" containerID="9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802" exitCode=0
Oct 30 22:16:27.093742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.093556    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" event={"ID":"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd","Type":"ContainerDied","Data":"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"}
Oct 30 22:16:27.093804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.093778    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn" event={"ID":"713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd","Type":"ContainerDied","Data":"49d3dfe4f586861dac4bcd228a0f7670c9b80f3399671052aba3626a72562c14"}
Oct 30 22:16:27.093858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.093635    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"
Oct 30 22:16:27.093906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.093807    4882 scope.go:122] "RemoveContainer" containerID="9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"
Oct 30 22:16:27.105897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.105610    4882 scope.go:122] "RemoveContainer" containerID="9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"
Oct 30 22:16:27.106013 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:27.105954    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802\": container with ID starting with 9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802 not found: ID does not exist" containerID="9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"
Oct 30 22:16:27.106013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.105996    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802"} err="failed to get container status \"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802\": rpc error: code = NotFound desc = could not find container \"9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802\": container with ID starting with 9a46f9b2aa60543aa500eb8271dd43085d71bc93d6d91aabc28337fb1cf62802 not found: ID does not exist"
Oct 30 22:16:27.127440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.127414    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:16:27.161888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.161842    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6g9h\" (UniqueName: \"kubernetes.io/projected/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-kube-api-access-r6g9h\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.161888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.161882    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.263368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.263246    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\" (UniqueName: \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\") pod \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") "
Oct 30 22:16:27.263655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.263397    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\") pod \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\" (UID: \"dbc916b9-d00a-495a-993d-2c77bb1cd2a3\") "
Oct 30 22:16:27.263914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.263877    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume" pod "dbc916b9-d00a-495a-993d-2c77bb1cd2a3" (UID: "dbc916b9-d00a-495a-993d-2c77bb1cd2a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:27.265949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.265924    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw" pod "dbc916b9-d00a-495a-993d-2c77bb1cd2a3" (UID: "dbc916b9-d00a-495a-993d-2c77bb1cd2a3"). InnerVolumeSpecName "kube-api-access-nz6qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:27.365655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.365578    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nz6qw\" (UniqueName: \"kubernetes.io/projected/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-kube-api-access-nz6qw\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.365655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.365656    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/dbc916b9-d00a-495a-993d-2c77bb1cd2a3-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:27.383330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.383296    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:16:27.389385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:27.389356    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-664cc54b9-r2tvn"]
Oct 30 22:16:28.101010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.100978    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"
Oct 30 22:16:28.101511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.101011    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z" event={"ID":"dbc916b9-d00a-495a-993d-2c77bb1cd2a3","Type":"ContainerDied","Data":"34a3eca544a7e277d824a5b3265d432e2c608e8805ec7f2679c28de6b825fb7e"}
Oct 30 22:16:28.101511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.101080    4882 scope.go:122] "RemoveContainer" containerID="8642a72afdb6d080255d24346944885535d6c12b51ca0625bbffb623e5c21736"
Oct 30 22:16:28.272186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.272150    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:16:28.320377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.320339    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-138566-7d49cb8c99-sjw5z"]
Oct 30 22:16:28.944153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.944118    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" path="/var/lib/kubelet/pods/713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd/volumes"
Oct 30 22:16:28.944567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:28.944553    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" path="/var/lib/kubelet/pods/dbc916b9-d00a-495a-993d-2c77bb1cd2a3/volumes"
Oct 30 22:16:45.759427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:45.759385    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:16:45.871366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:45.871330    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:16:50.760017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:50.759953    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" podUID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" containerName="payload-processing" containerID="cri-o://f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0" gracePeriod=25
Oct 30 22:16:50.808720 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:50.808677    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:16:50.872668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:50.872617    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" podUID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" containerName="payload-pre-processing" containerID="cri-o://efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e" gracePeriod=25
Oct 30 22:16:51.035608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.035336    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:16:51.139579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.139551    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\" (UniqueName: \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\") pod \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") "
Oct 30 22:16:51.139722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.139680    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\") pod \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\" (UID: \"1166b5f1-3b27-4c4e-a837-ad17bd1822b2\") "
Oct 30 22:16:51.140040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.140012    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume" pod "1166b5f1-3b27-4c4e-a837-ad17bd1822b2" (UID: "1166b5f1-3b27-4c4e-a837-ad17bd1822b2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:51.140201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.140189    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.141690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.141668    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v" pod "1166b5f1-3b27-4c4e-a837-ad17bd1822b2" (UID: "1166b5f1-3b27-4c4e-a837-ad17bd1822b2"). InnerVolumeSpecName "kube-api-access-psp5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:51.159139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.159093    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:16:51.241159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.241124    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\") pod \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") "
Oct 30 22:16:51.241339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.241216    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\" (UniqueName: \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\") pod \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\" (UID: \"9a5c22a7-1601-431e-b7cf-1f140bcd0066\") "
Oct 30 22:16:51.241541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.241518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume" pod "9a5c22a7-1601-431e-b7cf-1f140bcd0066" (UID: "9a5c22a7-1601-431e-b7cf-1f140bcd0066"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:16:51.241632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.241543    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psp5v\" (UniqueName: \"kubernetes.io/projected/1166b5f1-3b27-4c4e-a837-ad17bd1822b2-kube-api-access-psp5v\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.243431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.243411    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46" pod "9a5c22a7-1601-431e-b7cf-1f140bcd0066" (UID: "9a5c22a7-1601-431e-b7cf-1f140bcd0066"). InnerVolumeSpecName "kube-api-access-h4m46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:16:51.259181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.259148    4882 generic.go:356] "Generic (PLEG): container finished" podID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" containerID="efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e" exitCode=0
Oct 30 22:16:51.259324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.259206    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"
Oct 30 22:16:51.259324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.259230    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" event={"ID":"9a5c22a7-1601-431e-b7cf-1f140bcd0066","Type":"ContainerDied","Data":"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"}
Oct 30 22:16:51.259324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.259284    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l" event={"ID":"9a5c22a7-1601-431e-b7cf-1f140bcd0066","Type":"ContainerDied","Data":"a5ab06daecb54d92c696b026509e7d829bf0c8a93954c9ddcc9a6ceb06223fef"}
Oct 30 22:16:51.259324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.259305    4882 scope.go:122] "RemoveContainer" containerID="efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"
Oct 30 22:16:51.260572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.260548    4882 generic.go:356] "Generic (PLEG): container finished" podID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" containerID="f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0" exitCode=0
Oct 30 22:16:51.260694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.260630    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"
Oct 30 22:16:51.260752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.260631    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" event={"ID":"1166b5f1-3b27-4c4e-a837-ad17bd1822b2","Type":"ContainerDied","Data":"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"}
Oct 30 22:16:51.260752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.260731    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb" event={"ID":"1166b5f1-3b27-4c4e-a837-ad17bd1822b2","Type":"ContainerDied","Data":"18b3bbc61ca400baeb2d3ff12a179737cc352440a2eae735fb44ca74274aa0e4"}
Oct 30 22:16:51.271133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.270823    4882 scope.go:122] "RemoveContainer" containerID="efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"
Oct 30 22:16:51.271244 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:51.271217    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e\": container with ID starting with efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e not found: ID does not exist" containerID="efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"
Oct 30 22:16:51.271292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.271251    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e"} err="failed to get container status \"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e\": rpc error: code = NotFound desc = could not find container \"efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e\": container with ID starting with efada8450d14aaee363f201208231066b71921d083000b99ed9ddfc9851dcc0e not found: ID does not exist"
Oct 30 22:16:51.271292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.271273    4882 scope.go:122] "RemoveContainer" containerID="f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"
Oct 30 22:16:51.283743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.282927    4882 scope.go:122] "RemoveContainer" containerID="f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"
Oct 30 22:16:51.286967 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:16:51.286931    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0\": container with ID starting with f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0 not found: ID does not exist" containerID="f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"
Oct 30 22:16:51.287171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.286978    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0"} err="failed to get container status \"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0\": rpc error: code = NotFound desc = could not find container \"f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0\": container with ID starting with f2d4dbc6f21dd61441afea885f36340c54b9f652cb9039241c7b20254e05dbc0 not found: ID does not exist"
Oct 30 22:16:51.343093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.343049    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4m46\" (UniqueName: \"kubernetes.io/projected/9a5c22a7-1601-431e-b7cf-1f140bcd0066-kube-api-access-h4m46\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.343093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.343092    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a5c22a7-1601-431e-b7cf-1f140bcd0066-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:16:51.393554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.393513    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:16:51.401728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.401698    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-76cdc9998c-zgb6l"]
Oct 30 22:16:51.515214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.515167    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:16:51.517877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:51.517838    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-fe745d-5db6565677-b4rwb"]
Oct 30 22:16:52.944011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:52.943972    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" path="/var/lib/kubelet/pods/1166b5f1-3b27-4c4e-a837-ad17bd1822b2/volumes"
Oct 30 22:16:52.944486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:52.944385    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" path="/var/lib/kubelet/pods/9a5c22a7-1601-431e-b7cf-1f140bcd0066/volumes"
Oct 30 22:16:58.562608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:58.562560    4882 scope.go:122] "RemoveContainer" containerID="d3342c87b050601f0741121e4719f5e2a46530679a1c04f33f32b1fd8ae42e3d"
Oct 30 22:16:59.165614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.165568    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:16:59.166203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166183    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166205    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166225    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166231    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166241    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166251    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166268    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166276    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166285    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166292    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166303    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166310    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166327    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0eced86d-2e75-48f9-a98f-218e397abc97" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166334    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0eced86d-2e75-48f9-a98f-218e397abc97" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166345    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166352    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166363    4882 cpu_manager.go:414] "containerMap: removing container" podUID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166370    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166380    4882 cpu_manager.go:414] "containerMap: removing container" podUID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166387    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166396    4882 cpu_manager.go:414] "containerMap: removing container" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166403    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166416    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166425    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166438    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166445    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166455    4882 cpu_manager.go:414] "containerMap: removing container" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166463    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166475    4882 cpu_manager.go:414] "containerMap: removing container" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166484    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166500    4882 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166507    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166517    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166524    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166537    4882 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166543    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166554    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166562    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166572    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166580    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166606    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166612    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" containerName="payload-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166631    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3a02045c-9b27-4263-a592-1dc5793f6123" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166638    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a02045c-9b27-4263-a592-1dc5793f6123" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166648    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166655    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166684    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166692    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166703    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166711    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" containerName="payload-pre-processing"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166723    4882 cpu_manager.go:414] "containerMap: removing container" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166731    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166743    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166750    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166760    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166767    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166779    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166785    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166800    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166806    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166821    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166833    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166843    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.166940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166850    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.166993    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0eced86d-2e75-48f9-a98f-218e397abc97" containerName="payload-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167010    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ba52d04-6f37-4a45-90ca-0e85ea2a660c" containerName="payload-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167021    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a42d1c-be1b-4a3f-aff3-5db39f0fbb1a" containerName="payload-pre-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167033    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="713a6c2c-afec-41e9-afb8-7c3cbe4ac1bd" containerName="payload-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167042    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167052    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167061    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167068    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167077    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167086    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167095    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167103    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167112    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167120    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9a5c22a7-1601-431e-b7cf-1f140bcd0066" containerName="payload-pre-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167129    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1166b5f1-3b27-4c4e-a837-ad17bd1822b2" containerName="payload-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167139    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a02045c-9b27-4263-a592-1dc5793f6123" containerName="payload-pre-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167149    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167158    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167165    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167175    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167186    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167194    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167202    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbc916b9-d00a-495a-993d-2c77bb1cd2a3" containerName="payload-pre-processing"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167212    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167222    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167230    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167237    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167249    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1b01b4df-6b9c-4d13-a74d-b9731cf66cec" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167259    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167269    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167279    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.169330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.167288    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2251a9a9-4960-4128-b081-c22ff5f560bc" containerName="cleanup"
Oct 30 22:16:59.173123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.173097    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.224329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.224293    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:16:59.270978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.270941    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-c4bfac\""
Oct 30 22:16:59.270978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.270963    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-c4bfac-dockercfg-k29nw\""
Oct 30 22:16:59.338700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.338665    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.338912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.338728    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqqt7\" (UniqueName: \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.440298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.440202    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.440298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.440272    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqqt7\" (UniqueName: \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.441040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.441013    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.502993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.502958    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqqt7\" (UniqueName: \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\") pod \"payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.787041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.786955    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:16:59.961963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.961925    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:16:59.962783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962755    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.962783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962780    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962794    4882 cpu_manager.go:414] "containerMap: removing container" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962802    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54245ea6-5e26-4b20-b2d8-870dbdb1a180" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962814    4882 cpu_manager.go:414] "containerMap: removing container" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962821    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44e73d51-1357-476b-a4c7-29f3c0a121c9" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962840    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962846    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962859    4882 cpu_manager.go:414] "containerMap: removing container" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962865    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962876    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.962965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.962884    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963028    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2f96be1-7f98-4480-9459-b3411fa98c5c" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963045    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="29fdc1d3-ba80-461d-aa21-2db22628dc33" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963055    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c5569be-cfec-43bc-ac28-538fc7de0f11" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963065    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e58a043-0b72-4471-8694-b7480513764b" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963077    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad3c024c-9c76-4caa-9136-44d3eebaf342" containerName="cleanup"
Oct 30 22:16:59.963428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.963087    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac7e81ef-6600-4b3b-870d-8e8d828d15a1" containerName="cleanup"
Oct 30 22:16:59.967717 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:16:59.967696    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.021558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.021527    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:17:00.050732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.050399    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.050732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.050457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gq7k9\" (UniqueName: \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.152128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.152090    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.152319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.152137    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gq7k9\" (UniqueName: \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.153112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.153086    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.198560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.198245    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:17:00.217458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.217420    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gq7k9\" (UniqueName: \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\") pod \"payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.283650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.283617    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:00.323806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.323719    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" event={"ID":"3dd7965b-0479-490a-b431-e0f21b568b85","Type":"ContainerStarted","Data":"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"}
Oct 30 22:17:00.323806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.323788    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:17:00.324010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.323808    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" event={"ID":"3dd7965b-0479-490a-b431-e0f21b568b85","Type":"ContainerStarted","Data":"10de6f4a86c77e5b517d6853b25070b69c944c0ba39f016076949cc5cefea141"}
Oct 30 22:17:00.446736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.446681    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" podStartSLOduration=1.446665341 podStartE2EDuration="1.446665341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:00.443487403 +0000 UTC m=+2343.616811827" watchObservedRunningTime="2026-10-30 22:17:00.446665341 +0000 UTC m=+2343.619989768"
Oct 30 22:17:00.661485 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:00.661428    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb7b6a60a_e9ba_4ffc_95d5_3b863387545f.slice/crio-14ea9eb16ae5a2ec22fb0f2633035478e99b262583ddab8bbb61e2c9f44d25f1 WatchSource:0}: Error finding container 14ea9eb16ae5a2ec22fb0f2633035478e99b262583ddab8bbb61e2c9f44d25f1: Status 404 returned error can't find the container with id 14ea9eb16ae5a2ec22fb0f2633035478e99b262583ddab8bbb61e2c9f44d25f1
Oct 30 22:17:00.664398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:00.664370    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:17:01.332479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:01.332425    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" event={"ID":"b7b6a60a-e9ba-4ffc-95d5-3b863387545f","Type":"ContainerStarted","Data":"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"}
Oct 30 22:17:01.332925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:01.332493    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" event={"ID":"b7b6a60a-e9ba-4ffc-95d5-3b863387545f","Type":"ContainerStarted","Data":"14ea9eb16ae5a2ec22fb0f2633035478e99b262583ddab8bbb61e2c9f44d25f1"}
Oct 30 22:17:01.332925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:01.332753    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:01.452809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:01.452752    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" podStartSLOduration=2.452735957 podStartE2EDuration="2.452735957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:16:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:01.449939614 +0000 UTC m=+2344.623264052" watchObservedRunningTime="2026-10-30 22:17:01.452735957 +0000 UTC m=+2344.626060384"
Oct 30 22:17:02.108361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:02.108328    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"]
Oct 30 22:17:02.111844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:02.111816    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-a3aa65-hsvsv"]
Oct 30 22:17:02.943115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:02.943079    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cacf374-f0be-4fdb-8200-d770a322bfd1" path="/var/lib/kubelet/pods/4cacf374-f0be-4fdb-8200-d770a322bfd1/volumes"
Oct 30 22:17:11.335502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:11.335424    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:17:12.340284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:12.340253    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:17:13.459293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.459256    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:17:13.463197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.463178    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.514146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.514105    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.514338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.514156    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9gbk\" (UniqueName: \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.594120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.594082    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-5d2b13-dockercfg-zk4pw\""
Oct 30 22:17:13.594311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.594144    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-5d2b13\""
Oct 30 22:17:13.597128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.597096    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:17:13.616072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.616037    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.616250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.616081    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9gbk\" (UniqueName: \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.616873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.616846    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.688067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.688031    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9gbk\" (UniqueName: \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\") pod \"payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:13.777253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:13.777168    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:14.152483 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:14.152446    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbcfcebd_be25_4c03_8462_77dbd40c55ab.slice/crio-a1fd9e43291021f7d629603bb9332ee56c21eb7e8a4f243d26958706b0c19b9e WatchSource:0}: Error finding container a1fd9e43291021f7d629603bb9332ee56c21eb7e8a4f243d26958706b0c19b9e: Status 404 returned error can't find the container with id a1fd9e43291021f7d629603bb9332ee56c21eb7e8a4f243d26958706b0c19b9e
Oct 30 22:17:14.155823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.155797    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:17:14.275042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.274995    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:17:14.279034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.279016    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.326558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.326525    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.326756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.326687    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l5cn\" (UniqueName: \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.336037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.336005    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:17:14.409363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.409262    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:17:14.412496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.412467    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" event={"ID":"bbcfcebd-be25-4c03-8462-77dbd40c55ab","Type":"ContainerStarted","Data":"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"}
Oct 30 22:17:14.412682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.412518    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" event={"ID":"bbcfcebd-be25-4c03-8462-77dbd40c55ab","Type":"ContainerStarted","Data":"a1fd9e43291021f7d629603bb9332ee56c21eb7e8a4f243d26958706b0c19b9e"}
Oct 30 22:17:14.412682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.412558    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:14.428104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.428071    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.428260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.428240    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l5cn\" (UniqueName: \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.428863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.428845    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.490954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.490920    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l5cn\" (UniqueName: \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\") pod \"payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.533819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.533746    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" podStartSLOduration=1.533723868 podStartE2EDuration="1.533723868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:14.529762865 +0000 UTC m=+2357.703087294" watchObservedRunningTime="2026-10-30 22:17:14.533723868 +0000 UTC m=+2357.707048296"
Oct 30 22:17:14.593247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.593207    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:14.976174 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:14.976137    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0a3688_b947_4437_82f4_2b87428bfc4e.slice/crio-d61de3bc3953547c1d1876a39539348b3406e1f362ad8521347275926ce0a80d WatchSource:0}: Error finding container d61de3bc3953547c1d1876a39539348b3406e1f362ad8521347275926ce0a80d: Status 404 returned error can't find the container with id d61de3bc3953547c1d1876a39539348b3406e1f362ad8521347275926ce0a80d
Oct 30 22:17:14.978525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:14.978498    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:17:15.420367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:15.420321    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" event={"ID":"2e0a3688-b947-4437-82f4-2b87428bfc4e","Type":"ContainerStarted","Data":"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"}
Oct 30 22:17:15.420631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:15.420390    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" event={"ID":"2e0a3688-b947-4437-82f4-2b87428bfc4e","Type":"ContainerStarted","Data":"d61de3bc3953547c1d1876a39539348b3406e1f362ad8521347275926ce0a80d"}
Oct 30 22:17:15.420631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:15.420561    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:15.541917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:15.541857    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" podStartSLOduration=1.541836381 podStartE2EDuration="1.541836381s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:15.538274052 +0000 UTC m=+2358.711598480" watchObservedRunningTime="2026-10-30 22:17:15.541836381 +0000 UTC m=+2358.715160808"
Oct 30 22:17:19.600885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.600849    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"]
Oct 30 22:17:19.605071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.605045    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.661319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.661261    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"]
Oct 30 22:17:19.693342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.693301    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.693531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.693383    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.693609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.693579    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xwm5x\" (UniqueName: \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.707809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.707770    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:19.707809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.707798    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:17:19.794986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.794955    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xwm5x\" (UniqueName: \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.795163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.795017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.795163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.795076    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.795786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.795763    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.797468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.797441    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.857565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.857526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xwm5x\" (UniqueName: \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\") pod \"maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:19.920052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:19.920013    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:20.295078 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:20.295046    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6201c2c0_0539_47ff_8c23_4a0e7dd1ea8c.slice/crio-dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c WatchSource:0}: Error finding container dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c: Status 404 returned error can't find the container with id dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c
Oct 30 22:17:20.297840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:20.297814    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"]
Oct 30 22:17:20.454010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:20.453975    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz" event={"ID":"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c","Type":"ContainerStarted","Data":"dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c"}
Oct 30 22:17:20.812009 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:17:20.811923    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:17:21.461494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:21.461454    4882 generic.go:356] "Generic (PLEG): container finished" podID="6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" containerID="9a8ce8a87a868bfb0c505085725faa6e27803017c13c90fd1eb347df320844ee" exitCode=0
Oct 30 22:17:21.461718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:21.461544    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz" event={"ID":"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c","Type":"ContainerDied","Data":"9a8ce8a87a868bfb0c505085725faa6e27803017c13c90fd1eb347df320844ee"}
Oct 30 22:17:22.610630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.610578    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:22.728656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.728528    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\") pod \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") "
Oct 30 22:17:22.728822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.728746    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\" (UniqueName: \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\") pod \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") "
Oct 30 22:17:22.728881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.728870    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\") pod \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\" (UID: \"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c\") "
Oct 30 22:17:22.732346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.730019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca" pod "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" (UID: "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:22.732520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.732448    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca" pod "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" (UID: "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:22.732638 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.732572    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x" pod "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" (UID: "6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c"). InnerVolumeSpecName "kube-api-access-xwm5x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:22.829946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.829907    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:22.829946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.829935    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:22.829946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:22.829947    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xwm5x\" (UniqueName: \"kubernetes.io/projected/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c-kube-api-access-xwm5x\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:23.474616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:23.474549    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz" event={"ID":"6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c","Type":"ContainerDied","Data":"dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c"}
Oct 30 22:17:23.474616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:23.474576    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"
Oct 30 22:17:23.474893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:23.474624    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dbf5aac78a85bab9948fb7b61eb1eb889970254ee24e997423dde59c65bc643c"
Oct 30 22:17:25.423473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:25.423441    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:17:26.428784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:26.428754    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:17:35.865490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.865455    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:17:35.866176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.866156    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" containerName="revoke-keys"
Oct 30 22:17:35.866262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.866178    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" containerName="revoke-keys"
Oct 30 22:17:35.866342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.866329    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" containerName="revoke-keys"
Oct 30 22:17:35.870985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.870961    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:35.925778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.925747    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:17:35.972218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.972178    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-138566\""
Oct 30 22:17:35.972401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.972286    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-138566-dockercfg-9d8ld\""
Oct 30 22:17:35.981831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.981796    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:35.982024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:35.982003    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26bq6\" (UniqueName: \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.083286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.083243    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26bq6\" (UniqueName: \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.083493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.083377    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.084191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.084167    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.169689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.169612    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26bq6\" (UniqueName: \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\") pod \"payload-processing-e2e-ipp-b-138566-869577984b-lbbd7\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.184913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.184880    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:36.565841 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:36.565806    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cd86a48_01c0_4fcc_8514_c26842ad7518.slice/crio-8a58bd5eeefab4fe5da137f98fe6df75b26befe8bc422f195bcd9f3b56465ee6 WatchSource:0}: Error finding container 8a58bd5eeefab4fe5da137f98fe6df75b26befe8bc422f195bcd9f3b56465ee6: Status 404 returned error can't find the container with id 8a58bd5eeefab4fe5da137f98fe6df75b26befe8bc422f195bcd9f3b56465ee6
Oct 30 22:17:36.569672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:36.569644    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:17:37.558599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:37.558547    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" event={"ID":"5cd86a48-01c0-4fcc-8514-c26842ad7518","Type":"ContainerStarted","Data":"0d2c686cb1c26aad8f0ffd94dad22568f9f0874d96d5c44acd5c48f3d5e0f3fc"}
Oct 30 22:17:37.559081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:37.558620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" event={"ID":"5cd86a48-01c0-4fcc-8514-c26842ad7518","Type":"ContainerStarted","Data":"8a58bd5eeefab4fe5da137f98fe6df75b26befe8bc422f195bcd9f3b56465ee6"}
Oct 30 22:17:37.559081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:37.558651    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:37.679665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:37.679578    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" podStartSLOduration=2.679555198 podStartE2EDuration="2.679555198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:37.677889589 +0000 UTC m=+2380.851214017" watchObservedRunningTime="2026-10-30 22:17:37.679555198 +0000 UTC m=+2380.852879627"
Oct 30 22:17:45.066965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:45.066938    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz_2e0a3688-b947-4437-82f4-2b87428bfc4e/payload-processing/0.log"
Oct 30 22:17:46.280850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.280811    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"]
Oct 30 22:17:46.285340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.285315    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.339189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.339153    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"]
Oct 30 22:17:46.386114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.386077    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:17:46.386114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.386077    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:17:46.408928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.408886    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfn9f\" (UniqueName: \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.409108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.409004    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.409185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.409165    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.510513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.510464    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.510730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.510566    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.510730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.510666    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfn9f\" (UniqueName: \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.511300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.511279    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.512982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.512964    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.572444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.572375    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfn9f\" (UniqueName: \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\") pod \"maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.599164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.599127    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:46.978369 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:46.978334    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd16d3381_866f_4c37_9566_9dab73aca39a.slice/crio-75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff WatchSource:0}: Error finding container 75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff: Status 404 returned error can't find the container with id 75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff
Oct 30 22:17:46.980874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:46.980845    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"]
Oct 30 22:17:47.623601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:47.623551    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2" event={"ID":"d16d3381-866f-4c37-9566-9dab73aca39a","Type":"ContainerStarted","Data":"9b47f7af9b6588a39464c03ddd3f5092a98f920a3c5482f12344afa3aa4aa613"}
Oct 30 22:17:47.624025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:47.623632    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2" event={"ID":"d16d3381-866f-4c37-9566-9dab73aca39a","Type":"ContainerStarted","Data":"75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff"}
Oct 30 22:17:47.741217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:47.741159    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2" podStartSLOduration=1.741144783 podStartE2EDuration="1.741144783s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:17:47.740121018 +0000 UTC m=+2390.913445446" watchObservedRunningTime="2026-10-30 22:17:47.741144783 +0000 UTC m=+2390.914469210"
Oct 30 22:17:48.568342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:48.568307    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:17:48.632346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:48.632310    4882 generic.go:356] "Generic (PLEG): container finished" podID="d16d3381-866f-4c37-9566-9dab73aca39a" containerID="9b47f7af9b6588a39464c03ddd3f5092a98f920a3c5482f12344afa3aa4aa613" exitCode=0
Oct 30 22:17:48.632770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:48.632397    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2" event={"ID":"d16d3381-866f-4c37-9566-9dab73aca39a","Type":"ContainerDied","Data":"9b47f7af9b6588a39464c03ddd3f5092a98f920a3c5482f12344afa3aa4aa613"}
Oct 30 22:17:49.825324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.825294    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:49.953354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.953253    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\") pod \"d16d3381-866f-4c37-9566-9dab73aca39a\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") "
Oct 30 22:17:49.953354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.953308    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\" (UniqueName: \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\") pod \"d16d3381-866f-4c37-9566-9dab73aca39a\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") "
Oct 30 22:17:49.953354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.953334    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\") pod \"d16d3381-866f-4c37-9566-9dab73aca39a\" (UID: \"d16d3381-866f-4c37-9566-9dab73aca39a\") "
Oct 30 22:17:49.953735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.953702    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca" pod "d16d3381-866f-4c37-9566-9dab73aca39a" (UID: "d16d3381-866f-4c37-9566-9dab73aca39a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:17:49.953906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.953882    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:49.955554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.955519    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca" pod "d16d3381-866f-4c37-9566-9dab73aca39a" (UID: "d16d3381-866f-4c37-9566-9dab73aca39a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:17:49.955705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:49.955596    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f" pod "d16d3381-866f-4c37-9566-9dab73aca39a" (UID: "d16d3381-866f-4c37-9566-9dab73aca39a"). InnerVolumeSpecName "kube-api-access-wfn9f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:17:50.055631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:50.055567    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wfn9f\" (UniqueName: \"kubernetes.io/projected/d16d3381-866f-4c37-9566-9dab73aca39a-kube-api-access-wfn9f\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:50.055631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:50.055629    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d16d3381-866f-4c37-9566-9dab73aca39a-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:17:50.646896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:50.646857    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2" event={"ID":"d16d3381-866f-4c37-9566-9dab73aca39a","Type":"ContainerDied","Data":"75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff"}
Oct 30 22:17:50.647084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:50.646933    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="75e4f583a8139657f69f12a45bab404051bc2e352434457ca95a09ac060e24ff"
Oct 30 22:17:50.647084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:50.646874    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"
Oct 30 22:17:50.811989 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:17:50.811947    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:17:57.265521 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:57.265487    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:17:57.276395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:57.276370    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:17:59.246393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.246353    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:17:59.246990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.246974    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d16d3381-866f-4c37-9566-9dab73aca39a" containerName="revoke-keys"
Oct 30 22:17:59.247037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.246991    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d16d3381-866f-4c37-9566-9dab73aca39a" containerName="revoke-keys"
Oct 30 22:17:59.247123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.247114    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d16d3381-866f-4c37-9566-9dab73aca39a" containerName="revoke-keys"
Oct 30 22:17:59.251125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.251101    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.260402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.260369    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.260545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.260521    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhxsh\" (UniqueName: \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.305916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.305881    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:17:59.352224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.352193    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-fe745d\""
Oct 30 22:17:59.352224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.352194    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-fe745d-dockercfg-vb2kk\""
Oct 30 22:17:59.361086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.361048    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhxsh\" (UniqueName: \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.361227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.361153    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.361842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.361823    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.422801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.422770    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhxsh\" (UniqueName: \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\") pod \"payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.565767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.565681    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:17:59.944359 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:17:59.944325    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08901306_e819_40a1_ad61_8329776fb5b8.slice/crio-faaea3ad47bed8e4ce901005319dd078b6952df2cede78c7fe3b38f56b4cc697 WatchSource:0}: Error finding container faaea3ad47bed8e4ce901005319dd078b6952df2cede78c7fe3b38f56b4cc697: Status 404 returned error can't find the container with id faaea3ad47bed8e4ce901005319dd078b6952df2cede78c7fe3b38f56b4cc697
Oct 30 22:17:59.945870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:17:59.945829    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:18:00.711973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:00.711935    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" event={"ID":"08901306-e819-40a1-ad61-8329776fb5b8","Type":"ContainerStarted","Data":"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"}
Oct 30 22:18:00.712383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:00.711992    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" event={"ID":"08901306-e819-40a1-ad61-8329776fb5b8","Type":"ContainerStarted","Data":"faaea3ad47bed8e4ce901005319dd078b6952df2cede78c7fe3b38f56b4cc697"}
Oct 30 22:18:00.712383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:00.712023    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:18:00.830074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:00.830018    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" podStartSLOduration=1.82999994 podStartE2EDuration="1.82999994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:17:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:00.829718958 +0000 UTC m=+2404.003043387" watchObservedRunningTime="2026-10-30 22:18:00.82999994 +0000 UTC m=+2404.003324366"
Oct 30 22:18:04.812378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.812345    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"]
Oct 30 22:18:04.816690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.816667    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:04.870474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.870437    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"]
Oct 30 22:18:04.917279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.917243    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:04.917479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.917387    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:18:04.923143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.923103    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:04.923328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.923250    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:04.923328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:04.923283    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cwsx\" (UniqueName: \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.024917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.024876    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.025129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.024977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.025129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.025005    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cwsx\" (UniqueName: \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.025709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.025687    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.027342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.027313    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.087204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.087168    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cwsx\" (UniqueName: \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\") pod \"maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.131266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.131220    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:05.504457 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:18:05.504423    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7615d73f_3745_4101_b387_45d3c84b05cb.slice/crio-eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55 WatchSource:0}: Error finding container eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55: Status 404 returned error can't find the container with id eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55
Oct 30 22:18:05.514885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.514857    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"]
Oct 30 22:18:05.745189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:05.745143    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll" event={"ID":"7615d73f-3745-4101-b387-45d3c84b05cb","Type":"ContainerStarted","Data":"eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55"}
Oct 30 22:18:06.752554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:06.752463    4882 generic.go:356] "Generic (PLEG): container finished" podID="7615d73f-3745-4101-b387-45d3c84b05cb" containerID="2a5d79dc589070edf3b5938fb6297d39cb0b5d42fcbab2aebbe28084c12aff8e" exitCode=0
Oct 30 22:18:06.753096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:06.752550    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll" event={"ID":"7615d73f-3745-4101-b387-45d3c84b05cb","Type":"ContainerDied","Data":"2a5d79dc589070edf3b5938fb6297d39cb0b5d42fcbab2aebbe28084c12aff8e"}
Oct 30 22:18:07.920485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.920457    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:07.965884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.965845    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\" (UniqueName: \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\") pod \"7615d73f-3745-4101-b387-45d3c84b05cb\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") "
Oct 30 22:18:07.966065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.965947    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\") pod \"7615d73f-3745-4101-b387-45d3c84b05cb\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") "
Oct 30 22:18:07.966065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.966020    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\") pod \"7615d73f-3745-4101-b387-45d3c84b05cb\" (UID: \"7615d73f-3745-4101-b387-45d3c84b05cb\") "
Oct 30 22:18:07.966383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.966353    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca" pod "7615d73f-3745-4101-b387-45d3c84b05cb" (UID: "7615d73f-3745-4101-b387-45d3c84b05cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:07.968050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.968018    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx" pod "7615d73f-3745-4101-b387-45d3c84b05cb" (UID: "7615d73f-3745-4101-b387-45d3c84b05cb"). InnerVolumeSpecName "kube-api-access-6cwsx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:07.968050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:07.968035    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca" pod "7615d73f-3745-4101-b387-45d3c84b05cb" (UID: "7615d73f-3745-4101-b387-45d3c84b05cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:08.068269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.068174    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cwsx\" (UniqueName: \"kubernetes.io/projected/7615d73f-3745-4101-b387-45d3c84b05cb-kube-api-access-6cwsx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:08.068269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.068210    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:08.068269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.068224    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7615d73f-3745-4101-b387-45d3c84b05cb-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:08.765979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.765935    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll" event={"ID":"7615d73f-3745-4101-b387-45d3c84b05cb","Type":"ContainerDied","Data":"eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55"}
Oct 30 22:18:08.766164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.765992    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eee0e8f8b165138927a5d6b38ffa15689a2a65078b355c2d929c89b32af3dd55"
Oct 30 22:18:08.766164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:08.765957    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"
Oct 30 22:18:10.164526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.164485    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:18:10.165214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.164739    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" podUID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" containerName="payload-pre-processing" containerID="cri-o://28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf" gracePeriod=30
Oct 30 22:18:10.441786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.441755    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:18:10.492105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.492068    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\" (UniqueName: \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\") pod \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") "
Oct 30 22:18:10.492302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.492169    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\") pod \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\" (UID: \"acf9268b-77ed-491b-bb2e-ab0f95f64b53\") "
Oct 30 22:18:10.492573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.492549    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume" pod "acf9268b-77ed-491b-bb2e-ab0f95f64b53" (UID: "acf9268b-77ed-491b-bb2e-ab0f95f64b53"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:10.492724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.492703    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acf9268b-77ed-491b-bb2e-ab0f95f64b53-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:10.494265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.494235    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk" pod "acf9268b-77ed-491b-bb2e-ab0f95f64b53" (UID: "acf9268b-77ed-491b-bb2e-ab0f95f64b53"). InnerVolumeSpecName "kube-api-access-27jtk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:10.594355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.594317    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jtk\" (UniqueName: \"kubernetes.io/projected/acf9268b-77ed-491b-bb2e-ab0f95f64b53-kube-api-access-27jtk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:10.779915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.779821    4882 generic.go:356] "Generic (PLEG): container finished" podID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" containerID="28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf" exitCode=0
Oct 30 22:18:10.779915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.779867    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"
Oct 30 22:18:10.779915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.779901    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" event={"ID":"acf9268b-77ed-491b-bb2e-ab0f95f64b53","Type":"ContainerDied","Data":"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"}
Oct 30 22:18:10.780147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.779955    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs" event={"ID":"acf9268b-77ed-491b-bb2e-ab0f95f64b53","Type":"ContainerDied","Data":"abc71de0916f4e2d1461baeab457ec5fe2c5b6de2c8f5dfdebba01acb211be00"}
Oct 30 22:18:10.780147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.779978    4882 scope.go:122] "RemoveContainer" containerID="28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"
Oct 30 22:18:10.790265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.790245    4882 scope.go:122] "RemoveContainer" containerID="28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"
Oct 30 22:18:10.790574 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:10.790555    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf\": container with ID starting with 28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf not found: ID does not exist" containerID="28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"
Oct 30 22:18:10.790682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.790601    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf"} err="failed to get container status \"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf\": rpc error: code = NotFound desc = could not find container \"28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf\": container with ID starting with 28f74178942ce87b81f280d14b13ab9ddee919b05ed9a072e4c791e21b2a53cf not found: ID does not exist"
Oct 30 22:18:10.948686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.948651    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:18:10.952232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:10.952201    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-a4257d-c48cd847-7s2vs"]
Oct 30 22:18:11.720980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:11.720946    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:18:11.765425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:11.765392    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"]
Oct 30 22:18:11.768315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:11.768290    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-a4257d-k7rf2"]
Oct 30 22:18:12.943450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:12.943415    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" path="/var/lib/kubelet/pods/acf9268b-77ed-491b-bb2e-ab0f95f64b53/volumes"
Oct 30 22:18:12.943892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:12.943877    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d16d3381-866f-4c37-9566-9dab73aca39a" path="/var/lib/kubelet/pods/d16d3381-866f-4c37-9566-9dab73aca39a/volumes"
Oct 30 22:18:18.210011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.209970    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:18:18.210499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.210176    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" podUID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" containerName="payload-processing" containerID="cri-o://0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02" gracePeriod=30
Oct 30 22:18:18.263688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.263654    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:18:18.263899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.263862    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" podUID="3e124c0b-7950-436d-a84b-ca91bd1babf2" containerName="payload-pre-processing" containerID="cri-o://f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9" gracePeriod=30
Oct 30 22:18:18.533148 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.533115    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:18:18.575821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.575796    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:18:18.692133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692063    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\" (UniqueName: \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\") pod \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") "
Oct 30 22:18:18.692133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692138    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\") pod \"3e124c0b-7950-436d-a84b-ca91bd1babf2\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") "
Oct 30 22:18:18.692418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692159    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\") pod \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\" (UID: \"20c77d7d-7635-4a9f-972a-2c7b8533ae16\") "
Oct 30 22:18:18.692418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692277    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\" (UniqueName: \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\") pod \"3e124c0b-7950-436d-a84b-ca91bd1babf2\" (UID: \"3e124c0b-7950-436d-a84b-ca91bd1babf2\") "
Oct 30 22:18:18.692559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692527    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume" pod "3e124c0b-7950-436d-a84b-ca91bd1babf2" (UID: "3e124c0b-7950-436d-a84b-ca91bd1babf2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:18.692871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692845    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume" pod "20c77d7d-7635-4a9f-972a-2c7b8533ae16" (UID: "20c77d7d-7635-4a9f-972a-2c7b8533ae16"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:18.692989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692952    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e124c0b-7950-436d-a84b-ca91bd1babf2-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:18.692989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.692967    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/20c77d7d-7635-4a9f-972a-2c7b8533ae16-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:18.694473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.694451    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp" pod "20c77d7d-7635-4a9f-972a-2c7b8533ae16" (UID: "20c77d7d-7635-4a9f-972a-2c7b8533ae16"). InnerVolumeSpecName "kube-api-access-kwkwp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:18.694649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.694610    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67" pod "3e124c0b-7950-436d-a84b-ca91bd1babf2" (UID: "3e124c0b-7950-436d-a84b-ca91bd1babf2"). InnerVolumeSpecName "kube-api-access-5hr67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:18.794762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.794674    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kwkwp\" (UniqueName: \"kubernetes.io/projected/20c77d7d-7635-4a9f-972a-2c7b8533ae16-kube-api-access-kwkwp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:18.794762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.794715    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hr67\" (UniqueName: \"kubernetes.io/projected/3e124c0b-7950-436d-a84b-ca91bd1babf2-kube-api-access-5hr67\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:18.833175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.833142    4882 generic.go:356] "Generic (PLEG): container finished" podID="3e124c0b-7950-436d-a84b-ca91bd1babf2" containerID="f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9" exitCode=0
Oct 30 22:18:18.833385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.833190    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"
Oct 30 22:18:18.833385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.833231    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" event={"ID":"3e124c0b-7950-436d-a84b-ca91bd1babf2","Type":"ContainerDied","Data":"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"}
Oct 30 22:18:18.833385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.833291    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp" event={"ID":"3e124c0b-7950-436d-a84b-ca91bd1babf2","Type":"ContainerDied","Data":"a2fe1f629ae332c66795bf9509626b3b35ddae882cba9eb56deef71c5e6f9b47"}
Oct 30 22:18:18.833385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.833320    4882 scope.go:122] "RemoveContainer" containerID="f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"
Oct 30 22:18:18.834732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.834708    4882 generic.go:356] "Generic (PLEG): container finished" podID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" containerID="0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02" exitCode=0
Oct 30 22:18:18.834840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.834766    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"
Oct 30 22:18:18.834840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.834797    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" event={"ID":"20c77d7d-7635-4a9f-972a-2c7b8533ae16","Type":"ContainerDied","Data":"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"}
Oct 30 22:18:18.834953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.834846    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j" event={"ID":"20c77d7d-7635-4a9f-972a-2c7b8533ae16","Type":"ContainerDied","Data":"5b8466076653ce0622ff21d065163b7542928d095931a5c640c4e1283505b4f6"}
Oct 30 22:18:18.844808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.844784    4882 scope.go:122] "RemoveContainer" containerID="f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"
Oct 30 22:18:18.845118 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:18.845073    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9\": container with ID starting with f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9 not found: ID does not exist" containerID="f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"
Oct 30 22:18:18.845118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.845099    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9"} err="failed to get container status \"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9\": rpc error: code = NotFound desc = could not find container \"f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9\": container with ID starting with f9da9dd0e63d32a21fce006f782a9e36efe8c7ffd35a32fe22c90b0e3ab53ca9 not found: ID does not exist"
Oct 30 22:18:18.845118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.845116    4882 scope.go:122] "RemoveContainer" containerID="0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"
Oct 30 22:18:18.854301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.854278    4882 scope.go:122] "RemoveContainer" containerID="0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"
Oct 30 22:18:18.854546 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:18.854527    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02\": container with ID starting with 0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02 not found: ID does not exist" containerID="0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"
Oct 30 22:18:18.854620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:18.854554    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02"} err="failed to get container status \"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02\": rpc error: code = NotFound desc = could not find container \"0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02\": container with ID starting with 0aad340e9c7a10f833a1c07676ace79a6cafa2dfadbc6ced1b9142e56f618b02 not found: ID does not exist"
Oct 30 22:18:19.006682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:19.006644    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:18:19.010176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:19.010152    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-95b84c-5459cb9787-q2thp"]
Oct 30 22:18:19.136487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:19.136451    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:18:19.140911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:19.140885    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-95b84c-5dd7c4f69b-9hh5j"]
Oct 30 22:18:20.437246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.437206    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"]
Oct 30 22:18:20.487433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.487401    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-95b84c-88wll"]
Oct 30 22:18:20.809385 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:20.809292    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:18:20.815716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.815687    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"]
Oct 30 22:18:20.816443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816424    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3e124c0b-7950-436d-a84b-ca91bd1babf2" containerName="payload-pre-processing"
Oct 30 22:18:20.816503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816445    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3e124c0b-7950-436d-a84b-ca91bd1babf2" containerName="payload-pre-processing"
Oct 30 22:18:20.816503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816472    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" containerName="payload-processing"
Oct 30 22:18:20.816503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816477    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" containerName="payload-processing"
Oct 30 22:18:20.816503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816495    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7615d73f-3745-4101-b387-45d3c84b05cb" containerName="revoke-keys"
Oct 30 22:18:20.816503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816500    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7615d73f-3745-4101-b387-45d3c84b05cb" containerName="revoke-keys"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816515    4882 cpu_manager.go:414] "containerMap: removing container" podUID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" containerName="payload-pre-processing"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816520    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" containerName="payload-pre-processing"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816618    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="acf9268b-77ed-491b-bb2e-ab0f95f64b53" containerName="payload-pre-processing"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816631    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3e124c0b-7950-436d-a84b-ca91bd1babf2" containerName="payload-pre-processing"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816641    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" containerName="payload-processing"
Oct 30 22:18:20.816684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.816649    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7615d73f-3745-4101-b387-45d3c84b05cb" containerName="revoke-keys"
Oct 30 22:18:20.821511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.821488    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:20.876390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.876351    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"]
Oct 30 22:18:20.917575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.917531    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x94zp\" (UniqueName: \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:20.917803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.917780    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:20.917882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.917837    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:20.923545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.923439    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:20.923545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.923513    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:18:20.943241 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.943213    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20c77d7d-7635-4a9f-972a-2c7b8533ae16" path="/var/lib/kubelet/pods/20c77d7d-7635-4a9f-972a-2c7b8533ae16/volumes"
Oct 30 22:18:20.943653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.943639    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3e124c0b-7950-436d-a84b-ca91bd1babf2" path="/var/lib/kubelet/pods/3e124c0b-7950-436d-a84b-ca91bd1babf2/volumes"
Oct 30 22:18:20.944040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:20.944028    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7615d73f-3745-4101-b387-45d3c84b05cb" path="/var/lib/kubelet/pods/7615d73f-3745-4101-b387-45d3c84b05cb/volumes"
Oct 30 22:18:21.019154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.019114    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.019315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.019173    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.019315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.019251    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x94zp\" (UniqueName: \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.019945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.019918    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.021596 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.021570    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.081463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.081433    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x94zp\" (UniqueName: \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\") pod \"maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.135931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.135895    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:21.515178 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:18:21.515144    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75707616_642f_4953_99f9_84663ec8c012.slice/crio-291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197 WatchSource:0}: Error finding container 291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197: Status 404 returned error can't find the container with id 291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197
Oct 30 22:18:21.517748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.517724    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"]
Oct 30 22:18:21.862315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.862277    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4" event={"ID":"75707616-642f-4953-99f9-84663ec8c012","Type":"ContainerStarted","Data":"920fc4f6023cb91c5d0f01775bb8ab5eb1ff72a3ad551240196d108de3724f62"}
Oct 30 22:18:21.862471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.862334    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4" event={"ID":"75707616-642f-4953-99f9-84663ec8c012","Type":"ContainerStarted","Data":"291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197"}
Oct 30 22:18:21.981513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:21.981453    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4" podStartSLOduration=1.9814394960000001 podStartE2EDuration="1.981439496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:21.978961631 +0000 UTC m=+2425.152286059" watchObservedRunningTime="2026-10-30 22:18:21.981439496 +0000 UTC m=+2425.154763922"
Oct 30 22:18:22.871894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:22.871860    4882 generic.go:356] "Generic (PLEG): container finished" podID="75707616-642f-4953-99f9-84663ec8c012" containerID="920fc4f6023cb91c5d0f01775bb8ab5eb1ff72a3ad551240196d108de3724f62" exitCode=0
Oct 30 22:18:22.872376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:22.871946    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4" event={"ID":"75707616-642f-4953-99f9-84663ec8c012","Type":"ContainerDied","Data":"920fc4f6023cb91c5d0f01775bb8ab5eb1ff72a3ad551240196d108de3724f62"}
Oct 30 22:18:23.459171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:23.459138    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:18:24.032378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.032353    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:24.155957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.155873    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\") pod \"75707616-642f-4953-99f9-84663ec8c012\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") "
Oct 30 22:18:24.156130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.155979    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\" (UniqueName: \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\") pod \"75707616-642f-4953-99f9-84663ec8c012\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") "
Oct 30 22:18:24.156130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.156040    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\") pod \"75707616-642f-4953-99f9-84663ec8c012\" (UID: \"75707616-642f-4953-99f9-84663ec8c012\") "
Oct 30 22:18:24.156766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.156736    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca" pod "75707616-642f-4953-99f9-84663ec8c012" (UID: "75707616-642f-4953-99f9-84663ec8c012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:24.159154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.159123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp" pod "75707616-642f-4953-99f9-84663ec8c012" (UID: "75707616-642f-4953-99f9-84663ec8c012"). InnerVolumeSpecName "kube-api-access-x94zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:24.159281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.159250    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca" pod "75707616-642f-4953-99f9-84663ec8c012" (UID: "75707616-642f-4953-99f9-84663ec8c012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:24.257784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.257750    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/75707616-642f-4953-99f9-84663ec8c012-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:24.257784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.257780    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x94zp\" (UniqueName: \"kubernetes.io/projected/75707616-642f-4953-99f9-84663ec8c012-kube-api-access-x94zp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:24.257784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.257792    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/75707616-642f-4953-99f9-84663ec8c012-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:24.885968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.885938    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"
Oct 30 22:18:24.886190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.885980    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4" event={"ID":"75707616-642f-4953-99f9-84663ec8c012","Type":"ContainerDied","Data":"291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197"}
Oct 30 22:18:24.886190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:24.886035    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="291179cf744f07a75708399d3088d58e6188585a2b631eb88af680e84b3f4197"
Oct 30 22:18:25.597287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.597250    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"]
Oct 30 22:18:25.597943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.597925    4882 cpu_manager.go:414] "containerMap: removing container" podUID="75707616-642f-4953-99f9-84663ec8c012" containerName="revoke-keys"
Oct 30 22:18:25.597995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.597943    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75707616-642f-4953-99f9-84663ec8c012" containerName="revoke-keys"
Oct 30 22:18:25.598078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.598067    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="75707616-642f-4953-99f9-84663ec8c012" containerName="revoke-keys"
Oct 30 22:18:25.601801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.601774    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.656788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.656756    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"]
Oct 30 22:18:25.676394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.676363    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.676565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.676420    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfcrh\" (UniqueName: \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.676636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.676570    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.702791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.702756    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:25.702791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.702798    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:18:25.777237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.777203    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.777425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.777260    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfcrh\" (UniqueName: \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.777425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.777334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.778056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.778036    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.779694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.779671    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.839160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.839128    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfcrh\" (UniqueName: \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\") pod \"maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:25.916533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:25.916445    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:26.291818 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:18:26.291786    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33545c13_f344_4598_9b59_2e055c8068d1.slice/crio-0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33 WatchSource:0}: Error finding container 0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33: Status 404 returned error can't find the container with id 0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33
Oct 30 22:18:26.295114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:26.295087    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"]
Oct 30 22:18:26.899358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:26.899321    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42" event={"ID":"33545c13-f344-4598-9b59-2e055c8068d1","Type":"ContainerStarted","Data":"d22d42091da4ed506017c53fb106338fd8111866666a87064fae6c5ffef732c1"}
Oct 30 22:18:26.899818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:26.899381    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42" event={"ID":"33545c13-f344-4598-9b59-2e055c8068d1","Type":"ContainerStarted","Data":"0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33"}
Oct 30 22:18:27.018360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:27.018296    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42" podStartSLOduration=2.018281435 podStartE2EDuration="2.018281435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:27.016193579 +0000 UTC m=+2430.189518006" watchObservedRunningTime="2026-10-30 22:18:27.018281435 +0000 UTC m=+2430.191605864"
Oct 30 22:18:27.906660 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:27.906580    4882 generic.go:356] "Generic (PLEG): container finished" podID="33545c13-f344-4598-9b59-2e055c8068d1" containerID="d22d42091da4ed506017c53fb106338fd8111866666a87064fae6c5ffef732c1" exitCode=0
Oct 30 22:18:27.906660 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:27.906639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42" event={"ID":"33545c13-f344-4598-9b59-2e055c8068d1","Type":"ContainerDied","Data":"d22d42091da4ed506017c53fb106338fd8111866666a87064fae6c5ffef732c1"}
Oct 30 22:18:28.455546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.455510    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"]
Oct 30 22:18:28.460534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.460511    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.512851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.512797    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.513218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.513179    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-655t7\" (UniqueName: \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.513343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.513331    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.520080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.520053    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"]
Oct 30 22:18:28.614712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.614673    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.614886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.614762    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-655t7\" (UniqueName: \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.614886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.614857    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.615560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.615537    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.617142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.617115    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.676721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.676688    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-655t7\" (UniqueName: \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\") pod \"maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:28.775501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:28.775414    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:29.066640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.066614    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:29.120417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.120373    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\") pod \"33545c13-f344-4598-9b59-2e055c8068d1\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") "
Oct 30 22:18:29.120612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.120481    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\" (UniqueName: \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\") pod \"33545c13-f344-4598-9b59-2e055c8068d1\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") "
Oct 30 22:18:29.120700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.120650    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\") pod \"33545c13-f344-4598-9b59-2e055c8068d1\" (UID: \"33545c13-f344-4598-9b59-2e055c8068d1\") "
Oct 30 22:18:29.120815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.120784    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca" pod "33545c13-f344-4598-9b59-2e055c8068d1" (UID: "33545c13-f344-4598-9b59-2e055c8068d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:29.121081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.121063    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:29.122571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.122546    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh" pod "33545c13-f344-4598-9b59-2e055c8068d1" (UID: "33545c13-f344-4598-9b59-2e055c8068d1"). InnerVolumeSpecName "kube-api-access-kfcrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:29.122801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.122783    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca" pod "33545c13-f344-4598-9b59-2e055c8068d1" (UID: "33545c13-f344-4598-9b59-2e055c8068d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:29.151193 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:18:29.151157    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6b1ce6cd_7cfa_4502_94b9_bb1be6548972.slice/crio-361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237 WatchSource:0}: Error finding container 361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237: Status 404 returned error can't find the container with id 361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237
Oct 30 22:18:29.152887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.152854    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"]
Oct 30 22:18:29.222378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.222343    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfcrh\" (UniqueName: \"kubernetes.io/projected/33545c13-f344-4598-9b59-2e055c8068d1-kube-api-access-kfcrh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:29.222378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.222374    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/33545c13-f344-4598-9b59-2e055c8068d1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:29.926384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.926280    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9" event={"ID":"6b1ce6cd-7cfa-4502-94b9-bb1be6548972","Type":"ContainerStarted","Data":"ecf24ddeb1b615025008e758ffc8c9761d79b5a02d72ce5a52ed29caeb69d8f5"}
Oct 30 22:18:29.926550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.926390    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9" event={"ID":"6b1ce6cd-7cfa-4502-94b9-bb1be6548972","Type":"ContainerStarted","Data":"361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237"}
Oct 30 22:18:29.928076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.928045    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42" event={"ID":"33545c13-f344-4598-9b59-2e055c8068d1","Type":"ContainerDied","Data":"0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33"}
Oct 30 22:18:29.928076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.928076    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"
Oct 30 22:18:29.928252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:29.928095    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f864132c159df99953788bf4b34fdae08263c3b428ce0703d1c2b86f5bacf33"
Oct 30 22:18:30.046381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:30.046330    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9" podStartSLOduration=2.046314067 podStartE2EDuration="2.046314067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:18:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:18:30.043282587 +0000 UTC m=+2433.216607017" watchObservedRunningTime="2026-10-30 22:18:30.046314067 +0000 UTC m=+2433.219638494"
Oct 30 22:18:30.935932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:30.935898    4882 generic.go:356] "Generic (PLEG): container finished" podID="6b1ce6cd-7cfa-4502-94b9-bb1be6548972" containerID="ecf24ddeb1b615025008e758ffc8c9761d79b5a02d72ce5a52ed29caeb69d8f5" exitCode=0
Oct 30 22:18:30.942786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:30.942756    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9" event={"ID":"6b1ce6cd-7cfa-4502-94b9-bb1be6548972","Type":"ContainerDied","Data":"ecf24ddeb1b615025008e758ffc8c9761d79b5a02d72ce5a52ed29caeb69d8f5"}
Oct 30 22:18:32.101115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.101090    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:32.155811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.155773    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\" (UniqueName: \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\") pod \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") "
Oct 30 22:18:32.155811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.155816    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\") pod \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") "
Oct 30 22:18:32.156075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.155858    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\") pod \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\" (UID: \"6b1ce6cd-7cfa-4502-94b9-bb1be6548972\") "
Oct 30 22:18:32.156265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.156239    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca" pod "6b1ce6cd-7cfa-4502-94b9-bb1be6548972" (UID: "6b1ce6cd-7cfa-4502-94b9-bb1be6548972"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:32.158032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.158006    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca" pod "6b1ce6cd-7cfa-4502-94b9-bb1be6548972" (UID: "6b1ce6cd-7cfa-4502-94b9-bb1be6548972"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:32.158120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.158104    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7" pod "6b1ce6cd-7cfa-4502-94b9-bb1be6548972" (UID: "6b1ce6cd-7cfa-4502-94b9-bb1be6548972"). InnerVolumeSpecName "kube-api-access-655t7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:32.257631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.257512    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:32.257631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.257547    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:32.257631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.257557    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-655t7\" (UniqueName: \"kubernetes.io/projected/6b1ce6cd-7cfa-4502-94b9-bb1be6548972-kube-api-access-655t7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:32.965442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.965409    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"
Oct 30 22:18:32.965662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.965431    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9" event={"ID":"6b1ce6cd-7cfa-4502-94b9-bb1be6548972","Type":"ContainerDied","Data":"361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237"}
Oct 30 22:18:32.965662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:32.965495    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="361e9787ce2f14d463d1dcccc8396323db0860ce0d5a8a65ed20e6388d242237"
Oct 30 22:18:40.696001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:40.695912    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:18:40.696671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:40.696173    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" podUID="e7663bf3-860d-4b41-9bed-3794c2399651" containerName="payload-processing" containerID="cri-o://354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7" gracePeriod=30
Oct 30 22:18:40.811946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:40.811911    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:18:40.812385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:40.812349    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" podUID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" containerName="payload-pre-processing" containerID="cri-o://736f4b85d483882b109e7bee5a5ce5f5c2a683af80dbb51f49acb8aa7b1ebe0e" gracePeriod=30
Oct 30 22:18:40.987480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:40.987162    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:18:41.017578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.017547    4882 generic.go:356] "Generic (PLEG): container finished" podID="e7663bf3-860d-4b41-9bed-3794c2399651" containerID="354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7" exitCode=0
Oct 30 22:18:41.017756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.017628    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"
Oct 30 22:18:41.017756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.017639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" event={"ID":"e7663bf3-860d-4b41-9bed-3794c2399651","Type":"ContainerDied","Data":"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"}
Oct 30 22:18:41.017756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.017698    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5" event={"ID":"e7663bf3-860d-4b41-9bed-3794c2399651","Type":"ContainerDied","Data":"c09738cb5fabc92a789b6623086e4ef1c89ed54ee1cfba6a6478c8fa3569b5d4"}
Oct 30 22:18:41.017756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.017730    4882 scope.go:122] "RemoveContainer" containerID="354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"
Oct 30 22:18:41.021313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.021276    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" event={"ID":"edcb6f31-fe79-4529-8ccb-3141caeaf8c3","Type":"ContainerDied","Data":"736f4b85d483882b109e7bee5a5ce5f5c2a683af80dbb51f49acb8aa7b1ebe0e"}
Oct 30 22:18:41.021445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.021188    4882 generic.go:356] "Generic (PLEG): container finished" podID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" containerID="736f4b85d483882b109e7bee5a5ce5f5c2a683af80dbb51f49acb8aa7b1ebe0e" exitCode=0
Oct 30 22:18:41.030415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.030394    4882 scope.go:122] "RemoveContainer" containerID="354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"
Oct 30 22:18:41.031080 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:41.031040    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7\": container with ID starting with 354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7 not found: ID does not exist" containerID="354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"
Oct 30 22:18:41.031170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.031096    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7"} err="failed to get container status \"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7\": rpc error: code = NotFound desc = could not find container \"354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7\": container with ID starting with 354e309cff344997e5a18c65dda8b3f301c23140c340857234ae562bac0f28a7 not found: ID does not exist"
Oct 30 22:18:41.071777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.071736    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\" (UniqueName: \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\") pod \"e7663bf3-860d-4b41-9bed-3794c2399651\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") "
Oct 30 22:18:41.072002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.071980    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\") pod \"e7663bf3-860d-4b41-9bed-3794c2399651\" (UID: \"e7663bf3-860d-4b41-9bed-3794c2399651\") "
Oct 30 22:18:41.072573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.072546    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume" pod "e7663bf3-860d-4b41-9bed-3794c2399651" (UID: "e7663bf3-860d-4b41-9bed-3794c2399651"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.077768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.077739    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9" pod "e7663bf3-860d-4b41-9bed-3794c2399651" (UID: "e7663bf3-860d-4b41-9bed-3794c2399651"). InnerVolumeSpecName "kube-api-access-q9mv9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.095182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.095157    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:18:41.173412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.173381    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\") pod \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") "
Oct 30 22:18:41.173659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.173477    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\" (UniqueName: \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\") pod \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\" (UID: \"edcb6f31-fe79-4529-8ccb-3141caeaf8c3\") "
Oct 30 22:18:41.173837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.173823    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e7663bf3-860d-4b41-9bed-3794c2399651-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.173917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.173838    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q9mv9\" (UniqueName: \"kubernetes.io/projected/e7663bf3-860d-4b41-9bed-3794c2399651-kube-api-access-q9mv9\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.173917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.173862    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume" pod "edcb6f31-fe79-4529-8ccb-3141caeaf8c3" (UID: "edcb6f31-fe79-4529-8ccb-3141caeaf8c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:41.175557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.175536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272" pod "edcb6f31-fe79-4529-8ccb-3141caeaf8c3" (UID: "edcb6f31-fe79-4529-8ccb-3141caeaf8c3"). InnerVolumeSpecName "kube-api-access-2h272". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:41.275619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.275518    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.275619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.275551    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2h272\" (UniqueName: \"kubernetes.io/projected/edcb6f31-fe79-4529-8ccb-3141caeaf8c3-kube-api-access-2h272\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:41.441179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.441148    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:18:41.445344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.445314    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-f0b8c9-8596bff786-z4bv5"]
Oct 30 22:18:41.838348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.838313    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:18:41.838806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:41.838508    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" podUID="5cd86a48-01c0-4fcc-8514-c26842ad7518" containerName="payload-processing" containerID="cri-o://0d2c686cb1c26aad8f0ffd94dad22568f9f0874d96d5c44acd5c48f3d5e0f3fc" gracePeriod=30
Oct 30 22:18:42.037384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.037356    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"
Oct 30 22:18:42.037570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.037394    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r" event={"ID":"edcb6f31-fe79-4529-8ccb-3141caeaf8c3","Type":"ContainerDied","Data":"11acf7ddd348cce57710078b3d5248e0b10fee5ab4f0bc0861a6c2d8a62a1a28"}
Oct 30 22:18:42.037570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.037468    4882 scope.go:122] "RemoveContainer" containerID="736f4b85d483882b109e7bee5a5ce5f5c2a683af80dbb51f49acb8aa7b1ebe0e"
Oct 30 22:18:42.039700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.039518    4882 generic.go:356] "Generic (PLEG): container finished" podID="5cd86a48-01c0-4fcc-8514-c26842ad7518" containerID="0d2c686cb1c26aad8f0ffd94dad22568f9f0874d96d5c44acd5c48f3d5e0f3fc" exitCode=0
Oct 30 22:18:42.039700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.039599    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" event={"ID":"5cd86a48-01c0-4fcc-8514-c26842ad7518","Type":"ContainerDied","Data":"0d2c686cb1c26aad8f0ffd94dad22568f9f0874d96d5c44acd5c48f3d5e0f3fc"}
Oct 30 22:18:42.117507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.117481    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:18:42.185975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.185940    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\") pod \"5cd86a48-01c0-4fcc-8514-c26842ad7518\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") "
Oct 30 22:18:42.186138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.186034    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\" (UniqueName: \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\") pod \"5cd86a48-01c0-4fcc-8514-c26842ad7518\" (UID: \"5cd86a48-01c0-4fcc-8514-c26842ad7518\") "
Oct 30 22:18:42.186355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.186330    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume" pod "5cd86a48-01c0-4fcc-8514-c26842ad7518" (UID: "5cd86a48-01c0-4fcc-8514-c26842ad7518"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:42.188175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.188151    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6" pod "5cd86a48-01c0-4fcc-8514-c26842ad7518" (UID: "5cd86a48-01c0-4fcc-8514-c26842ad7518"). InnerVolumeSpecName "kube-api-access-26bq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:42.206299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.206270    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:18:42.210551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.210526    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-f0b8c9-c999d578c-dz97r"]
Oct 30 22:18:42.287267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.287181    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5cd86a48-01c0-4fcc-8514-c26842ad7518-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.287267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.287210    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26bq6\" (UniqueName: \"kubernetes.io/projected/5cd86a48-01c0-4fcc-8514-c26842ad7518-kube-api-access-26bq6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:42.343465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.343418    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"]
Oct 30 22:18:42.346965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.346933    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-f0b8c9-q89kz"]
Oct 30 22:18:42.945934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.945897    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c" path="/var/lib/kubelet/pods/6201c2c0-0539-47ff-8c23-4a0e7dd1ea8c/volumes"
Oct 30 22:18:42.946339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.946313    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7663bf3-860d-4b41-9bed-3794c2399651" path="/var/lib/kubelet/pods/e7663bf3-860d-4b41-9bed-3794c2399651/volumes"
Oct 30 22:18:42.946675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:42.946660    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" path="/var/lib/kubelet/pods/edcb6f31-fe79-4529-8ccb-3141caeaf8c3/volumes"
Oct 30 22:18:43.049475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.049446    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"
Oct 30 22:18:43.049475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.049463    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7" event={"ID":"5cd86a48-01c0-4fcc-8514-c26842ad7518","Type":"ContainerDied","Data":"8a58bd5eeefab4fe5da137f98fe6df75b26befe8bc422f195bcd9f3b56465ee6"}
Oct 30 22:18:43.049820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.049527    4882 scope.go:122] "RemoveContainer" containerID="0d2c686cb1c26aad8f0ffd94dad22568f9f0874d96d5c44acd5c48f3d5e0f3fc"
Oct 30 22:18:43.217434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.217350    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:18:43.267287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.267254    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-138566-869577984b-lbbd7"]
Oct 30 22:18:43.641112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.641082    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"]
Oct 30 22:18:43.643990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:43.643967    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-138566-gn8d9"]
Oct 30 22:18:44.943939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:44.943904    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cd86a48-01c0-4fcc-8514-c26842ad7518" path="/var/lib/kubelet/pods/5cd86a48-01c0-4fcc-8514-c26842ad7518/volumes"
Oct 30 22:18:44.944361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:44.944348    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6b1ce6cd-7cfa-4502-94b9-bb1be6548972" path="/var/lib/kubelet/pods/6b1ce6cd-7cfa-4502-94b9-bb1be6548972/volumes"
Oct 30 22:18:49.903811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.903773    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"]
Oct 30 22:18:49.904310 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904296    4882 cpu_manager.go:414] "containerMap: removing container" podUID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" containerName="payload-pre-processing"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904311    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" containerName="payload-pre-processing"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904325    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5cd86a48-01c0-4fcc-8514-c26842ad7518" containerName="payload-processing"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904331    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cd86a48-01c0-4fcc-8514-c26842ad7518" containerName="payload-processing"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904344    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6b1ce6cd-7cfa-4502-94b9-bb1be6548972" containerName="revoke-keys"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904350    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6b1ce6cd-7cfa-4502-94b9-bb1be6548972" containerName="revoke-keys"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904359    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e7663bf3-860d-4b41-9bed-3794c2399651" containerName="payload-processing"
Oct 30 22:18:49.904379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904364    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7663bf3-860d-4b41-9bed-3794c2399651" containerName="payload-processing"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904394    4882 cpu_manager.go:414] "containerMap: removing container" podUID="33545c13-f344-4598-9b59-2e055c8068d1" containerName="revoke-keys"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904399    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33545c13-f344-4598-9b59-2e055c8068d1" containerName="revoke-keys"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904479    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="33545c13-f344-4598-9b59-2e055c8068d1" containerName="revoke-keys"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904491    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="edcb6f31-fe79-4529-8ccb-3141caeaf8c3" containerName="payload-pre-processing"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904499    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cd86a48-01c0-4fcc-8514-c26842ad7518" containerName="payload-processing"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904507    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6b1ce6cd-7cfa-4502-94b9-bb1be6548972" containerName="revoke-keys"
Oct 30 22:18:49.904652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.904513    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7663bf3-860d-4b41-9bed-3794c2399651" containerName="payload-processing"
Oct 30 22:18:49.908178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.908150    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:49.961671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.961640    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:18:49.963930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:49.963900    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"]
Oct 30 22:18:50.009963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.009923    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:18:50.077141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.077104    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.077315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.077260    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.077315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.077283    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t55wl\" (UniqueName: \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.178164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.178073    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.178164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.178121    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t55wl\" (UniqueName: \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.178365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.178258    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.178941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.178912    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.180488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.180458    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.240643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.240609    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t55wl\" (UniqueName: \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\") pod \"maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.525163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.525066    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:50.808548 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:50.808452    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:18:50.901080 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:18:50.901042    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3edfd7a8_1792_44ed_a820_d2cfc87be180.slice/crio-a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504 WatchSource:0}: Error finding container a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504: Status 404 returned error can't find the container with id a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504
Oct 30 22:18:50.901814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:50.901789    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"]
Oct 30 22:18:51.103785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:51.103750    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw" event={"ID":"3edfd7a8-1792-44ed-a820-d2cfc87be180","Type":"ContainerStarted","Data":"a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504"}
Oct 30 22:18:52.112409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:52.112372    4882 generic.go:356] "Generic (PLEG): container finished" podID="3edfd7a8-1792-44ed-a820-d2cfc87be180" containerID="995776d5ab07648bee126574d3a2b6d45be6e7c3800339bf318285d7ef2d9c8b" exitCode=0
Oct 30 22:18:52.112879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:52.112464    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw" event={"ID":"3edfd7a8-1792-44ed-a820-d2cfc87be180","Type":"ContainerDied","Data":"995776d5ab07648bee126574d3a2b6d45be6e7c3800339bf318285d7ef2d9c8b"}
Oct 30 22:18:53.147342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.147304    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:18:53.147744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.147570    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing" containerID="cri-o://471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21" gracePeriod=30
Oct 30 22:18:53.256408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.256366    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:18:53.256677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.256646    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" podUID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" containerName="payload-pre-processing" containerID="cri-o://c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7" gracePeriod=30
Oct 30 22:18:53.368269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.366732    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:53.468512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.468481    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:18:53.527187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.527148    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\") pod \"3edfd7a8-1792-44ed-a820-d2cfc87be180\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") "
Oct 30 22:18:53.527363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.527349    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\" (UniqueName: \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\") pod \"3edfd7a8-1792-44ed-a820-d2cfc87be180\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") "
Oct 30 22:18:53.527455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.527414    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\") pod \"3edfd7a8-1792-44ed-a820-d2cfc87be180\" (UID: \"3edfd7a8-1792-44ed-a820-d2cfc87be180\") "
Oct 30 22:18:53.528272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.528242    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca" pod "3edfd7a8-1792-44ed-a820-d2cfc87be180" (UID: "3edfd7a8-1792-44ed-a820-d2cfc87be180"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:53.530251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.530222    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca" pod "3edfd7a8-1792-44ed-a820-d2cfc87be180" (UID: "3edfd7a8-1792-44ed-a820-d2cfc87be180"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:18:53.531567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.531534    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl" pod "3edfd7a8-1792-44ed-a820-d2cfc87be180" (UID: "3edfd7a8-1792-44ed-a820-d2cfc87be180"). InnerVolumeSpecName "kube-api-access-t55wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:53.547866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.547837    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:18:53.628502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.628462    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\") pod \"a26faa2b-bfec-4c15-9f2f-e029abf94670\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") "
Oct 30 22:18:53.628749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.628532    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\" (UniqueName: \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\") pod \"a26faa2b-bfec-4c15-9f2f-e029abf94670\" (UID: \"a26faa2b-bfec-4c15-9f2f-e029abf94670\") "
Oct 30 22:18:53.629128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.628933    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume" pod "a26faa2b-bfec-4c15-9f2f-e029abf94670" (UID: "a26faa2b-bfec-4c15-9f2f-e029abf94670"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:53.629128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.629053    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.629128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.629072    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a26faa2b-bfec-4c15-9f2f-e029abf94670-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.629128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.629089    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t55wl\" (UniqueName: \"kubernetes.io/projected/3edfd7a8-1792-44ed-a820-d2cfc87be180-kube-api-access-t55wl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.629128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.629102    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3edfd7a8-1792-44ed-a820-d2cfc87be180-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.631631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.631504    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8" pod "a26faa2b-bfec-4c15-9f2f-e029abf94670" (UID: "a26faa2b-bfec-4c15-9f2f-e029abf94670"). InnerVolumeSpecName "kube-api-access-f76d8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:53.730462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.730361    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\" (UniqueName: \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\") pod \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") "
Oct 30 22:18:53.730683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.730552    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\") pod \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\" (UID: \"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c\") "
Oct 30 22:18:53.731009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.730976    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f76d8\" (UniqueName: \"kubernetes.io/projected/a26faa2b-bfec-4c15-9f2f-e029abf94670-kube-api-access-f76d8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.731355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.731330    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume" pod "7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" (UID: "7ca187ec-f7a7-46fe-ad63-7812ddc13f2c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:53.733997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.733971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77" pod "7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" (UID: "7ca187ec-f7a7-46fe-ad63-7812ddc13f2c"). InnerVolumeSpecName "kube-api-access-k9l77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:53.736944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.736917    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:18:53.737146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.737125    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy" containerID="cri-o://5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562" gracePeriod=30
Oct 30 22:18:53.739975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.739948    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:18:53.740179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.740158    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy" containerID="cri-o://166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59" gracePeriod=30
Oct 30 22:18:53.832087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.832048    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9l77\" (UniqueName: \"kubernetes.io/projected/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-kube-api-access-k9l77\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:53.832087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:53.832087    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:54.127840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.127805    4882 generic.go:356] "Generic (PLEG): container finished" podID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" containerID="c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7" exitCode=0
Oct 30 22:18:54.128042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.127889    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"
Oct 30 22:18:54.128042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.127896    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" event={"ID":"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c","Type":"ContainerDied","Data":"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"}
Oct 30 22:18:54.128042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.127987    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4" event={"ID":"7ca187ec-f7a7-46fe-ad63-7812ddc13f2c","Type":"ContainerDied","Data":"24032ef9169ca4d34187b395ccf19f91d491d820c8d284d35432f0bb8ff9bd83"}
Oct 30 22:18:54.128042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.128017    4882 scope.go:122] "RemoveContainer" containerID="c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"
Oct 30 22:18:54.129576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.129561    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"
Oct 30 22:18:54.129736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.129697    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw" event={"ID":"3edfd7a8-1792-44ed-a820-d2cfc87be180","Type":"ContainerDied","Data":"a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504"}
Oct 30 22:18:54.129811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.129747    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4b3cd53791bf60ecdd8813bbb0cfd4fdfca7d62ee00dfbdada66e6b830db504"
Oct 30 22:18:54.132176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.132146    4882 generic.go:356] "Generic (PLEG): container finished" podID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerID="471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21" exitCode=0
Oct 30 22:18:54.132324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.132232    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"
Oct 30 22:18:54.132324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.132281    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerDied","Data":"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"}
Oct 30 22:18:54.132444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.132323    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7" event={"ID":"a26faa2b-bfec-4c15-9f2f-e029abf94670","Type":"ContainerDied","Data":"34a45ee0c491e8052098bb7be50233853d8057811bcdde7c66c7c10ba90789ac"}
Oct 30 22:18:54.141353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.141322    4882 scope.go:122] "RemoveContainer" containerID="c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"
Oct 30 22:18:54.142904 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:54.142875    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7\": container with ID starting with c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7 not found: ID does not exist" containerID="c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"
Oct 30 22:18:54.143010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.142915    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7"} err="failed to get container status \"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7\": rpc error: code = NotFound desc = could not find container \"c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7\": container with ID starting with c326ebd16d724550b9538d9bbeb50d8cef059c106c85006e59c4f936a903f9b7 not found: ID does not exist"
Oct 30 22:18:54.143010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.142938    4882 scope.go:122] "RemoveContainer" containerID="471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"
Oct 30 22:18:54.157776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.157749    4882 scope.go:122] "RemoveContainer" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"
Oct 30 22:18:54.180556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.180532    4882 scope.go:122] "RemoveContainer" containerID="471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"
Oct 30 22:18:54.180992 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:54.180951    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21\": container with ID starting with 471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21 not found: ID does not exist" containerID="471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"
Oct 30 22:18:54.181102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.180986    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21"} err="failed to get container status \"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21\": rpc error: code = NotFound desc = could not find container \"471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21\": container with ID starting with 471b2804d24677271bfa2a55ac6e7694e5fb61daae2628eef121d692f318ac21 not found: ID does not exist"
Oct 30 22:18:54.181102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.181010    4882 scope.go:122] "RemoveContainer" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"
Oct 30 22:18:54.181343 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:54.181319    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547\": container with ID starting with 3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547 not found: ID does not exist" containerID="3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"
Oct 30 22:18:54.181411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.181354    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547"} err="failed to get container status \"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547\": rpc error: code = NotFound desc = could not find container \"3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547\": container with ID starting with 3e9bc47e1c8d34ccea3068c788a7f5c9dbd81b9308fb0704468902730bb45547 not found: ID does not exist"
Oct 30 22:18:54.354016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.353979    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:18:54.358655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.358625    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-65659d-5f7d854c44-bbgk7"]
Oct 30 22:18:54.472260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.472177    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:18:54.476726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.476693    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-65659d-55c9cfbc9f-vxmf4"]
Oct 30 22:18:54.942948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.942916    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" path="/var/lib/kubelet/pods/7ca187ec-f7a7-46fe-ad63-7812ddc13f2c/volumes"
Oct 30 22:18:54.943340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:54.943327    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" path="/var/lib/kubelet/pods/a26faa2b-bfec-4c15-9f2f-e029abf94670/volumes"
Oct 30 22:18:55.018792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:55.018758    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"]
Oct 30 22:18:55.022852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:55.022828    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-65659d-g4f42"]
Oct 30 22:18:56.436181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:56.436145    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:18:56.945092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:56.945058    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33545c13-f344-4598-9b59-2e055c8068d1" path="/var/lib/kubelet/pods/33545c13-f344-4598-9b59-2e055c8068d1/volumes"
Oct 30 22:18:58.631321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:58.631288    4882 scope.go:122] "RemoveContainer" containerID="2f517e8715d00e2a02bbc9d40d3d9b66f4fe67bc2cd2937c6085c2d908f72f1b"
Oct 30 22:18:59.036412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.036374    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:18:59.061327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.061291    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:18:59.086805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.086771    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.086966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.086812    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.086966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.086836    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.086966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.086854    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.087110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.086991    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.087110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087062    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.087222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087109    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.087222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087150    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.087222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087158    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.087386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087221    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.087386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087238    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:59.087386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087290    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.087386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087351    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087451    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087496    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087504    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087551    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087578    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087614    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087642    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087704    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\") pod \"4ccb6660-3298-4d45-88c2-067273851619\" (UID: \"4ccb6660-3298-4d45-88c2-067273851619\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087725    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087734    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.087741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\") pod \"d971a92c-0cb0-42c0-87fe-1257153ba41d\" (UID: \"d971a92c-0cb0-42c0-87fe-1257153ba41d\") "
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088084    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088288    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d971a92c-0cb0-42c0-87fe-1257153ba41d-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088308    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4ccb6660-3298-4d45-88c2-067273851619-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088310    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088321    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088334    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088348    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088361    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088375    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.088796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.088561    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.089946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.089678    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.090123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.090103    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.090991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.090962    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "kube-api-access-4hhrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:59.090991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.090983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:59.091147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.091128    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token" pod "4ccb6660-3298-4d45-88c2-067273851619" (UID: "4ccb6660-3298-4d45-88c2-067273851619"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:59.091206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.091158    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "kube-api-access-lj4bs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:59.091282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.091260    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:18:59.091412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.091393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:18:59.091464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.091402    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy" pod "d971a92c-0cb0-42c0-87fe-1257153ba41d" (UID: "d971a92c-0cb0-42c0-87fe-1257153ba41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:18:59.171334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.171245    4882 generic.go:356] "Generic (PLEG): container finished" podID="4ccb6660-3298-4d45-88c2-067273851619" containerID="166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59" exitCode=0
Oct 30 22:18:59.171334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.171305    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"
Oct 30 22:18:59.171566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.171335    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" event={"ID":"4ccb6660-3298-4d45-88c2-067273851619","Type":"ContainerDied","Data":"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"}
Oct 30 22:18:59.171566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.171392    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5" event={"ID":"4ccb6660-3298-4d45-88c2-067273851619","Type":"ContainerDied","Data":"dcc9e0bc60c496d29397b9bf2d36a9f9002419b4806d083b89a52bc21d17058a"}
Oct 30 22:18:59.171566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.171420    4882 scope.go:122] "RemoveContainer" containerID="166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"
Oct 30 22:18:59.173196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.173172    4882 generic.go:356] "Generic (PLEG): container finished" podID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerID="5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562" exitCode=0
Oct 30 22:18:59.173401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.173242    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"
Oct 30 22:18:59.173401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.173268    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" event={"ID":"d971a92c-0cb0-42c0-87fe-1257153ba41d","Type":"ContainerDied","Data":"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"}
Oct 30 22:18:59.173401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.173331    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt" event={"ID":"d971a92c-0cb0-42c0-87fe-1257153ba41d","Type":"ContainerDied","Data":"cc7f457cfe8bfa15ff2caecb598af7e7956242a78f104a668603371ab7860e31"}
Oct 30 22:18:59.182641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.182614    4882 scope.go:122] "RemoveContainer" containerID="166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"
Oct 30 22:18:59.182980 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:59.182957    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59\": container with ID starting with 166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59 not found: ID does not exist" containerID="166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"
Oct 30 22:18:59.183064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.182988    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59"} err="failed to get container status \"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59\": rpc error: code = NotFound desc = could not find container \"166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59\": container with ID starting with 166a1acae082fbc23d3cba96fa62f39cb5b0f6aea5a3de2ceb68c2e51b038c59 not found: ID does not exist"
Oct 30 22:18:59.183064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.183006    4882 scope.go:122] "RemoveContainer" containerID="5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"
Oct 30 22:18:59.190018 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.189990    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190018 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190017    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190028    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hhrv\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-kube-api-access-4hhrv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190038    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190049    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4ccb6660-3298-4d45-88c2-067273851619-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190057    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190066    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190075    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj4bs\" (UniqueName: \"kubernetes.io/projected/d971a92c-0cb0-42c0-87fe-1257153ba41d-kube-api-access-lj4bs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190085    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d971a92c-0cb0-42c0-87fe-1257153ba41d-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190094    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4ccb6660-3298-4d45-88c2-067273851619-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.190189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.190102    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4ccb6660-3298-4d45-88c2-067273851619-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:18:59.192932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.192920    4882 scope.go:122] "RemoveContainer" containerID="5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"
Oct 30 22:18:59.193262 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:18:59.193240    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562\": container with ID starting with 5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562 not found: ID does not exist" containerID="5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"
Oct 30 22:18:59.193340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.193272    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562"} err="failed to get container status \"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562\": rpc error: code = NotFound desc = could not find container \"5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562\": container with ID starting with 5f02a7ae48a9317f2a09a8087a67b7454c87f978649f0afba309c71342641562 not found: ID does not exist"
Oct 30 22:18:59.324117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.324067    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:18:59.327389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.327362    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-wlnm5"]
Oct 30 22:18:59.506704 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.506617    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:18:59.511259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:18:59.511228    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksglt"]
Oct 30 22:19:00.600416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600383    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:19:00.600911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600895    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600912    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600922    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600928    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600946    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600951    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.600958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600957    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600961    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600971    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" containerName="payload-pre-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600975    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" containerName="payload-pre-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600995    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.600999    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601014    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3edfd7a8-1792-44ed-a820-d2cfc87be180" containerName="revoke-keys"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601018    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3edfd7a8-1792-44ed-a820-d2cfc87be180" containerName="revoke-keys"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601094    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601101    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601107    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ca187ec-f7a7-46fe-ad63-7812ddc13f2c" containerName="payload-pre-processing"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601117    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" containerName="istio-proxy"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601125    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3edfd7a8-1792-44ed-a820-d2cfc87be180" containerName="revoke-keys"
Oct 30 22:19:00.601153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.601132    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4ccb6660-3298-4d45-88c2-067273851619" containerName="istio-proxy"
Oct 30 22:19:00.606223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.606195    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.651182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.651150    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 22:19:00.651374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.651352    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" containerID="cri-o://73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8" gracePeriod=30
Oct 30 22:19:00.659237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.659208    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-3cecc7\""
Oct 30 22:19:00.659393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.659250    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-3cecc7-dockercfg-z6prm\""
Oct 30 22:19:00.659814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.659796    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:19:00.706803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.706766    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.706963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.706911    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vdr8\" (UniqueName: \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.732978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.732945    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:19:00.733558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.733544    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a26faa2b-bfec-4c15-9f2f-e029abf94670" containerName="payload-processing"
Oct 30 22:19:00.736921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.736903    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.791763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.791731    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:19:00.807844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.807811    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.808030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.807875    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vdr8\" (UniqueName: \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.808084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.808027    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4p7jt\" (UniqueName: \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.808155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.808092    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.809054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.809026    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.870068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.869991    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vdr8\" (UniqueName: \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\") pod \"payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.909282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.909236    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4p7jt\" (UniqueName: \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.909500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.909409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.910175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.910154    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:00.919996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.919969    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:00.947120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.947082    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4ccb6660-3298-4d45-88c2-067273851619" path="/var/lib/kubelet/pods/4ccb6660-3298-4d45-88c2-067273851619/volumes"
Oct 30 22:19:00.947932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.947907    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d971a92c-0cb0-42c0-87fe-1257153ba41d" path="/var/lib/kubelet/pods/d971a92c-0cb0-42c0-87fe-1257153ba41d/volumes"
Oct 30 22:19:00.970792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:00.970755    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4p7jt\" (UniqueName: \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\") pod \"payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:01.051157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:01.051120    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:01.295109 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:01.295071    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8dcac791_17ec_481d_8ce5_59ebbc9f94f1.slice/crio-f5e2fcad48ab9bc0eb6f6af3c995b1fc4d332adb8de137faab322ff4e3737ea7 WatchSource:0}: Error finding container f5e2fcad48ab9bc0eb6f6af3c995b1fc4d332adb8de137faab322ff4e3737ea7: Status 404 returned error can't find the container with id f5e2fcad48ab9bc0eb6f6af3c995b1fc4d332adb8de137faab322ff4e3737ea7
Oct 30 22:19:01.297747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:01.297718    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:19:01.426543 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:01.426506    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod825553dd_6d58_4bc7_a297_c5ae0a74647e.slice/crio-88e3f59a1004a2b3a9517dd64a80084696c07a26563c57e34b39d761d078f40b WatchSource:0}: Error finding container 88e3f59a1004a2b3a9517dd64a80084696c07a26563c57e34b39d761d078f40b: Status 404 returned error can't find the container with id 88e3f59a1004a2b3a9517dd64a80084696c07a26563c57e34b39d761d078f40b
Oct 30 22:19:01.428963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:01.428939    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:19:02.196186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.196140    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" event={"ID":"8dcac791-17ec-481d-8ce5-59ebbc9f94f1","Type":"ContainerStarted","Data":"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"}
Oct 30 22:19:02.196733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.196214    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" event={"ID":"8dcac791-17ec-481d-8ce5-59ebbc9f94f1","Type":"ContainerStarted","Data":"f5e2fcad48ab9bc0eb6f6af3c995b1fc4d332adb8de137faab322ff4e3737ea7"}
Oct 30 22:19:02.196733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.196286    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:02.197821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.197785    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" event={"ID":"825553dd-6d58-4bc7-a297-c5ae0a74647e","Type":"ContainerStarted","Data":"075be162225d24979bd5d7f4a88c83b7c9d116da66455f94a7eebc33939b32ef"}
Oct 30 22:19:02.197984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.197827    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" event={"ID":"825553dd-6d58-4bc7-a297-c5ae0a74647e","Type":"ContainerStarted","Data":"88e3f59a1004a2b3a9517dd64a80084696c07a26563c57e34b39d761d078f40b"}
Oct 30 22:19:02.197984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.197877    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:02.316396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.316342    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" podStartSLOduration=2.316324786 podStartE2EDuration="2.316324786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:02.315074889 +0000 UTC m=+2465.488399317" watchObservedRunningTime="2026-10-30 22:19:02.316324786 +0000 UTC m=+2465.489649213"
Oct 30 22:19:02.434242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.434186    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" podStartSLOduration=2.434170342 podStartE2EDuration="2.434170342s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:02.431679851 +0000 UTC m=+2465.605004278" watchObservedRunningTime="2026-10-30 22:19:02.434170342 +0000 UTC m=+2465.607494806"
Oct 30 22:19:02.490887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.490805    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:02.490887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:02.490858    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:05.930794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.930732    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 22:19:05.966661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966621    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.966824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966673    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.966824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966736    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.966824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966788    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966844    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966887    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966912    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966951    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966985    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\") pod \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\" (UID: \"53fcd401-e36f-45c3-b3c5-1af35be22bb7\") "
Oct 30 22:19:05.967033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.966990    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:05.967283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967166    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:05.967283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967184    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:05.967283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967265    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:05.967434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:05.968088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967611    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.968088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967640    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.968088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967656    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.968088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967672    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.968088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.967687    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:05.969410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.969382    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:05.969505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.969435    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "kube-api-access-j9lr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:05.969667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.969644    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:05.969765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:05.969667    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy" pod "53fcd401-e36f-45c3-b3c5-1af35be22bb7" (UID: "53fcd401-e36f-45c3-b3c5-1af35be22bb7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:06.068692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.068657    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.068692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.068686    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.068692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.068696    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/53fcd401-e36f-45c3-b3c5-1af35be22bb7-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.068935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.068707    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j9lr5\" (UniqueName: \"kubernetes.io/projected/53fcd401-e36f-45c3-b3c5-1af35be22bb7-kube-api-access-j9lr5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:06.073225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.073195    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:19:06.073409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.073388    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" containerID="cri-o://149f6d50b4a093a84de57301dc7e0019d509ed4530b051d70891b7a11647e1b5" gracePeriod=30
Oct 30 22:19:06.222914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.222833    4882 generic.go:356] "Generic (PLEG): container finished" podID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerID="73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8" exitCode=0
Oct 30 22:19:06.222914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.222887    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"
Oct 30 22:19:06.223115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.222921    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" event={"ID":"53fcd401-e36f-45c3-b3c5-1af35be22bb7","Type":"ContainerDied","Data":"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"}
Oct 30 22:19:06.223115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.222981    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk" event={"ID":"53fcd401-e36f-45c3-b3c5-1af35be22bb7","Type":"ContainerDied","Data":"855e0a3ae8e77cf3bc5e953cfdbe782be6a6f95b3b841cff631070d0944f1636"}
Oct 30 22:19:06.223115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.223005    4882 scope.go:122] "RemoveContainer" containerID="73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"
Oct 30 22:19:06.233722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.233701    4882 scope.go:122] "RemoveContainer" containerID="73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"
Oct 30 22:19:06.234016 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:06.233993    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8\": container with ID starting with 73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8 not found: ID does not exist" containerID="73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"
Oct 30 22:19:06.234063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.234026    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8"} err="failed to get container status \"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8\": rpc error: code = NotFound desc = could not find container \"73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8\": container with ID starting with 73ab893977503ee273f87137f7fc30fee697dd22951176a8deedc060870471c8 not found: ID does not exist"
Oct 30 22:19:06.358676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.358638    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 22:19:06.362470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.362444    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-5k9wk"]
Oct 30 22:19:06.783432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.783399    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:19:06.783598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.783464    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:19:06.944364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:06.944322    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" path="/var/lib/kubelet/pods/53fcd401-e36f-45c3-b3c5-1af35be22bb7/volumes"
Oct 30 22:19:11.260132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.260098    4882 generic.go:356] "Generic (PLEG): container finished" podID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerID="149f6d50b4a093a84de57301dc7e0019d509ed4530b051d70891b7a11647e1b5" exitCode=0
Oct 30 22:19:11.260579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.260169    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" event={"ID":"ae462eed-1dab-45f5-8af4-eeede962eea9","Type":"ContainerDied","Data":"149f6d50b4a093a84de57301dc7e0019d509ed4530b051d70891b7a11647e1b5"}
Oct 30 22:19:11.349925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.349859    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:19:11.393312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.393273    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:19:11.393491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.393471    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" podUID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" containerName="payload-processing" containerID="cri-o://ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692" gracePeriod=30
Oct 30 22:19:11.432388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432351    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432396    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432483    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432520    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432628    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432702    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432760    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432790    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.432857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432820    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\") pod \"ae462eed-1dab-45f5-8af4-eeede962eea9\" (UID: \"ae462eed-1dab-45f5-8af4-eeede962eea9\") "
Oct 30 22:19:11.433026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:11.433026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.432889    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:11.433269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.433143    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:11.433269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.433168    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:11.433579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.433539    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:11.434610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.434569    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.434724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.434616    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.434724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.434631    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.434724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.434646    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae462eed-1dab-45f5-8af4-eeede962eea9-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.434724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.434660    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.435768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.435149    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:19:11.435996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.435971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:11.436084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.436019    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "kube-api-access-flm86". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:11.436232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.436206    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy" pod "ae462eed-1dab-45f5-8af4-eeede962eea9" (UID: "ae462eed-1dab-45f5-8af4-eeede962eea9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:19:11.450814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.450774    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:19:11.451091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.451053    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" podUID="3dd7965b-0479-490a-b431-e0f21b568b85" containerName="payload-pre-processing" containerID="cri-o://454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99" gracePeriod=30
Oct 30 22:19:11.535605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.535562    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.535605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.535608    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.535798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.535624    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flm86\" (UniqueName: \"kubernetes.io/projected/ae462eed-1dab-45f5-8af4-eeede962eea9-kube-api-access-flm86\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.535798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.535637    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae462eed-1dab-45f5-8af4-eeede962eea9-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.714167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.714140    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:19:11.740904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.740877    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:19:11.838366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838333    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\" (UniqueName: \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\") pod \"3dd7965b-0479-490a-b431-e0f21b568b85\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") "
Oct 30 22:19:11.838569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838373    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\" (UniqueName: \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\") pod \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") "
Oct 30 22:19:11.838569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838505    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\") pod \"3dd7965b-0479-490a-b431-e0f21b568b85\" (UID: \"3dd7965b-0479-490a-b431-e0f21b568b85\") "
Oct 30 22:19:11.838569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838564    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\") pod \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\" (UID: \"b7b6a60a-e9ba-4ffc-95d5-3b863387545f\") "
Oct 30 22:19:11.838902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838880    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume" pod "3dd7965b-0479-490a-b431-e0f21b568b85" (UID: "3dd7965b-0479-490a-b431-e0f21b568b85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:11.838965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.838883    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume" pod "b7b6a60a-e9ba-4ffc-95d5-3b863387545f" (UID: "b7b6a60a-e9ba-4ffc-95d5-3b863387545f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:11.839040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.839025    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3dd7965b-0479-490a-b431-e0f21b568b85-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.839097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.839042    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.840545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.840528    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9" pod "b7b6a60a-e9ba-4ffc-95d5-3b863387545f" (UID: "b7b6a60a-e9ba-4ffc-95d5-3b863387545f"). InnerVolumeSpecName "kube-api-access-gq7k9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:11.840664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.840640    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7" pod "3dd7965b-0479-490a-b431-e0f21b568b85" (UID: "3dd7965b-0479-490a-b431-e0f21b568b85"). InnerVolumeSpecName "kube-api-access-wqqt7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:11.940232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.940150    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqqt7\" (UniqueName: \"kubernetes.io/projected/3dd7965b-0479-490a-b431-e0f21b568b85-kube-api-access-wqqt7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:11.940232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:11.940178    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gq7k9\" (UniqueName: \"kubernetes.io/projected/b7b6a60a-e9ba-4ffc-95d5-3b863387545f-kube-api-access-gq7k9\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:12.270288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.270189    4882 generic.go:356] "Generic (PLEG): container finished" podID="3dd7965b-0479-490a-b431-e0f21b568b85" containerID="454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99" exitCode=0
Oct 30 22:19:12.270288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.270245    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"
Oct 30 22:19:12.270858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.270284    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" event={"ID":"3dd7965b-0479-490a-b431-e0f21b568b85","Type":"ContainerDied","Data":"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"}
Oct 30 22:19:12.270858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.270343    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4" event={"ID":"3dd7965b-0479-490a-b431-e0f21b568b85","Type":"ContainerDied","Data":"10de6f4a86c77e5b517d6853b25070b69c944c0ba39f016076949cc5cefea141"}
Oct 30 22:19:12.270858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.270369    4882 scope.go:122] "RemoveContainer" containerID="454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.272003    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.272029    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5" event={"ID":"ae462eed-1dab-45f5-8af4-eeede962eea9","Type":"ContainerDied","Data":"c3b2cfc32d1bc7adbcd56caf5aabb3a9c6b4a46e22d4ba3d9deedd90df92c678"}
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.273550    4882 generic.go:356] "Generic (PLEG): container finished" podID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" containerID="ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692" exitCode=0
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.273625    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.273638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" event={"ID":"b7b6a60a-e9ba-4ffc-95d5-3b863387545f","Type":"ContainerDied","Data":"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"}
Oct 30 22:19:12.273699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.273687    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9" event={"ID":"b7b6a60a-e9ba-4ffc-95d5-3b863387545f","Type":"ContainerDied","Data":"14ea9eb16ae5a2ec22fb0f2633035478e99b262583ddab8bbb61e2c9f44d25f1"}
Oct 30 22:19:12.282598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.282399    4882 scope.go:122] "RemoveContainer" containerID="454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"
Oct 30 22:19:12.282998 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:12.282796    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99\": container with ID starting with 454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99 not found: ID does not exist" containerID="454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"
Oct 30 22:19:12.282998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.282837    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99"} err="failed to get container status \"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99\": rpc error: code = NotFound desc = could not find container \"454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99\": container with ID starting with 454a12f9a37ceb28de81957904350bbe70871c2eeac3e9a4437e6441927b6f99 not found: ID does not exist"
Oct 30 22:19:12.282998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.282863    4882 scope.go:122] "RemoveContainer" containerID="149f6d50b4a093a84de57301dc7e0019d509ed4530b051d70891b7a11647e1b5"
Oct 30 22:19:12.295678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.295649    4882 scope.go:122] "RemoveContainer" containerID="ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"
Oct 30 22:19:12.305041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.305012    4882 scope.go:122] "RemoveContainer" containerID="ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"
Oct 30 22:19:12.305385 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:12.305363    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692\": container with ID starting with ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692 not found: ID does not exist" containerID="ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"
Oct 30 22:19:12.305482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.305393    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692"} err="failed to get container status \"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692\": rpc error: code = NotFound desc = could not find container \"ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692\": container with ID starting with ec3d82c752443e77965f797c5e05bc983a458bb32a36d6403db8003f69661692 not found: ID does not exist"
Oct 30 22:19:12.438398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.438359    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:19:12.444020 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.443992    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-c4bfac-6785856954-ffst9"]
Oct 30 22:19:12.556635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.556537    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:19:12.560415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.560385    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-c4bfac-7f546785c5-6z6b4"]
Oct 30 22:19:12.587658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.587625    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:19:12.587854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.587832    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" podUID="2e0a3688-b947-4437-82f4-2b87428bfc4e" containerName="payload-processing" containerID="cri-o://5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb" gracePeriod=30
Oct 30 22:19:12.645353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.645318    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:19:12.645773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.645551    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" podUID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" containerName="payload-pre-processing" containerID="cri-o://f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4" gracePeriod=30
Oct 30 22:19:12.671418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.671392    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:19:12.682169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.682137    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-x8nz5"]
Oct 30 22:19:12.907854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.907826    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:19:12.935927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.935890    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:19:12.951330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.951291    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3dd7965b-0479-490a-b431-e0f21b568b85" path="/var/lib/kubelet/pods/3dd7965b-0479-490a-b431-e0f21b568b85/volumes"
Oct 30 22:19:12.951777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.951763    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" path="/var/lib/kubelet/pods/ae462eed-1dab-45f5-8af4-eeede962eea9/volumes"
Oct 30 22:19:12.952118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:12.952106    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" path="/var/lib/kubelet/pods/b7b6a60a-e9ba-4ffc-95d5-3b863387545f/volumes"
Oct 30 22:19:13.050982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.050952    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\" (UniqueName: \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\") pod \"2e0a3688-b947-4437-82f4-2b87428bfc4e\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") "
Oct 30 22:19:13.051189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.050999    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\") pod \"2e0a3688-b947-4437-82f4-2b87428bfc4e\" (UID: \"2e0a3688-b947-4437-82f4-2b87428bfc4e\") "
Oct 30 22:19:13.051189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051037    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\" (UniqueName: \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\") pod \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") "
Oct 30 22:19:13.051189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051161    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\") pod \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\" (UID: \"bbcfcebd-be25-4c03-8462-77dbd40c55ab\") "
Oct 30 22:19:13.051463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051428    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume" pod "2e0a3688-b947-4437-82f4-2b87428bfc4e" (UID: "2e0a3688-b947-4437-82f4-2b87428bfc4e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:13.051580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051484    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume" pod "bbcfcebd-be25-4c03-8462-77dbd40c55ab" (UID: "bbcfcebd-be25-4c03-8462-77dbd40c55ab"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:13.051664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051647    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2e0a3688-b947-4437-82f4-2b87428bfc4e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.051716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.051669    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbcfcebd-be25-4c03-8462-77dbd40c55ab-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.053830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.053808    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn" pod "2e0a3688-b947-4437-82f4-2b87428bfc4e" (UID: "2e0a3688-b947-4437-82f4-2b87428bfc4e"). InnerVolumeSpecName "kube-api-access-8l5cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:13.053933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.053839    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk" pod "bbcfcebd-be25-4c03-8462-77dbd40c55ab" (UID: "bbcfcebd-be25-4c03-8462-77dbd40c55ab"). InnerVolumeSpecName "kube-api-access-r9gbk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:13.099262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.099229    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"]
Oct 30 22:19:13.103205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.103177    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-c4bfac-h6ndw"]
Oct 30 22:19:13.152603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.152566    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l5cn\" (UniqueName: \"kubernetes.io/projected/2e0a3688-b947-4437-82f4-2b87428bfc4e-kube-api-access-8l5cn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.152789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.152627    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9gbk\" (UniqueName: \"kubernetes.io/projected/bbcfcebd-be25-4c03-8462-77dbd40c55ab-kube-api-access-r9gbk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:13.206225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.206195    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:19:13.206400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.206267    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:19:13.282882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.282843    4882 generic.go:356] "Generic (PLEG): container finished" podID="2e0a3688-b947-4437-82f4-2b87428bfc4e" containerID="5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb" exitCode=0
Oct 30 22:19:13.283343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.282927    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"
Oct 30 22:19:13.283343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.282938    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" event={"ID":"2e0a3688-b947-4437-82f4-2b87428bfc4e","Type":"ContainerDied","Data":"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"}
Oct 30 22:19:13.283343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.282997    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz" event={"ID":"2e0a3688-b947-4437-82f4-2b87428bfc4e","Type":"ContainerDied","Data":"d61de3bc3953547c1d1876a39539348b3406e1f362ad8521347275926ce0a80d"}
Oct 30 22:19:13.283343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.283029    4882 scope.go:122] "RemoveContainer" containerID="5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"
Oct 30 22:19:13.284573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.284456    4882 generic.go:356] "Generic (PLEG): container finished" podID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" containerID="f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4" exitCode=0
Oct 30 22:19:13.284573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.284530    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"
Oct 30 22:19:13.284573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.284547    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" event={"ID":"bbcfcebd-be25-4c03-8462-77dbd40c55ab","Type":"ContainerDied","Data":"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"}
Oct 30 22:19:13.284796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.284604    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g" event={"ID":"bbcfcebd-be25-4c03-8462-77dbd40c55ab","Type":"ContainerDied","Data":"a1fd9e43291021f7d629603bb9332ee56c21eb7e8a4f243d26958706b0c19b9e"}
Oct 30 22:19:13.295274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.295248    4882 scope.go:122] "RemoveContainer" containerID="5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"
Oct 30 22:19:13.295661 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:13.295636    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb\": container with ID starting with 5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb not found: ID does not exist" containerID="5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"
Oct 30 22:19:13.295783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.295672    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb"} err="failed to get container status \"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb\": rpc error: code = NotFound desc = could not find container \"5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb\": container with ID starting with 5238267d8658859b59ade885a2037addbee737db2d98e494ade257e781dabffb not found: ID does not exist"
Oct 30 22:19:13.295783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.295697    4882 scope.go:122] "RemoveContainer" containerID="f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"
Oct 30 22:19:13.305068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.305048    4882 scope.go:122] "RemoveContainer" containerID="f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"
Oct 30 22:19:13.305368 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:13.305348    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4\": container with ID starting with f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4 not found: ID does not exist" containerID="f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"
Oct 30 22:19:13.305427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.305381    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4"} err="failed to get container status \"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4\": rpc error: code = NotFound desc = could not find container \"f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4\": container with ID starting with f2fd7125604ae45524b511580e0fec606e4ac319cbaab4cb2730ea090770dab4 not found: ID does not exist"
Oct 30 22:19:13.552813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.552737    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:19:13.555784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.555751    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-5d2b13-54dc6f8fd7-kf98g"]
Oct 30 22:19:13.631210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631168    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"]
Oct 30 22:19:13.631746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631726    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" containerName="payload-processing"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631747    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" containerName="payload-processing"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631764    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631771    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631797    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" containerName="payload-pre-processing"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631805    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" containerName="payload-pre-processing"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631833    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2e0a3688-b947-4437-82f4-2b87428bfc4e" containerName="payload-processing"
Oct 30 22:19:13.631860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631841    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0a3688-b947-4437-82f4-2b87428bfc4e" containerName="payload-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631864    4882 cpu_manager.go:414] "containerMap: removing container" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631872    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631881    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3dd7965b-0479-490a-b431-e0f21b568b85" containerName="payload-pre-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.631888    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3dd7965b-0479-490a-b431-e0f21b568b85" containerName="payload-pre-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632017    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b7b6a60a-e9ba-4ffc-95d5-3b863387545f" containerName="payload-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632032    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae462eed-1dab-45f5-8af4-eeede962eea9" containerName="istio-proxy"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632044    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" containerName="payload-pre-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632057    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0a3688-b947-4437-82f4-2b87428bfc4e" containerName="payload-processing"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632067    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="53fcd401-e36f-45c3-b3c5-1af35be22bb7" containerName="istio-proxy"
Oct 30 22:19:13.632202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.632075    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3dd7965b-0479-490a-b431-e0f21b568b85" containerName="payload-pre-processing"
Oct 30 22:19:13.636955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.636930    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.668500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.668464    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:19:13.673954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.673930    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-5d2b13-65f7fdf4d6-mwhzz"]
Oct 30 22:19:13.690683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.690650    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"]
Oct 30 22:19:13.758697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.758664    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27s6n\" (UniqueName: \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.758870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.758835    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.758910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.758874    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.771250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.771215    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:19:13.771392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.771306    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:19:13.859619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.859570    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.859779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.859642    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.859779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.859692    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27s6n\" (UniqueName: \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.860219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.860193    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.862040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.862022    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.922164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.921479    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27s6n\" (UniqueName: \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\") pod \"maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:13.950792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:13.950760    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:14.322077 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:14.322020    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod904fa85f_1c68_4848_ad2b_fce99694c124.slice/crio-aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc WatchSource:0}: Error finding container aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc: Status 404 returned error can't find the container with id aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc
Oct 30 22:19:14.324195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.324169    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"]
Oct 30 22:19:14.667114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.667081    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"]
Oct 30 22:19:14.669484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.669459    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5d2b13-w6zr4"]
Oct 30 22:19:14.943838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.943756    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0a3688-b947-4437-82f4-2b87428bfc4e" path="/var/lib/kubelet/pods/2e0a3688-b947-4437-82f4-2b87428bfc4e/volumes"
Oct 30 22:19:14.944197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.944184    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3edfd7a8-1792-44ed-a820-d2cfc87be180" path="/var/lib/kubelet/pods/3edfd7a8-1792-44ed-a820-d2cfc87be180/volumes"
Oct 30 22:19:14.944575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.944562    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75707616-642f-4953-99f9-84663ec8c012" path="/var/lib/kubelet/pods/75707616-642f-4953-99f9-84663ec8c012/volumes"
Oct 30 22:19:14.945215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:14.945201    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bbcfcebd-be25-4c03-8462-77dbd40c55ab" path="/var/lib/kubelet/pods/bbcfcebd-be25-4c03-8462-77dbd40c55ab/volumes"
Oct 30 22:19:15.302043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:15.302012    4882 generic.go:356] "Generic (PLEG): container finished" podID="904fa85f-1c68-4848-ad2b-fce99694c124" containerID="0c8d2b4cd274d90e834611a7720c449e33e4a1e53761ef810db9be34be616193" exitCode=0
Oct 30 22:19:15.302220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:15.302109    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw" event={"ID":"904fa85f-1c68-4848-ad2b-fce99694c124","Type":"ContainerDied","Data":"0c8d2b4cd274d90e834611a7720c449e33e4a1e53761ef810db9be34be616193"}
Oct 30 22:19:15.302220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:15.302181    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw" event={"ID":"904fa85f-1c68-4848-ad2b-fce99694c124","Type":"ContainerStarted","Data":"aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc"}
Oct 30 22:19:16.447936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.447907    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:16.588350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.588314    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\") pod \"904fa85f-1c68-4848-ad2b-fce99694c124\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") "
Oct 30 22:19:16.588350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.588352    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\") pod \"904fa85f-1c68-4848-ad2b-fce99694c124\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") "
Oct 30 22:19:16.588570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.588390    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\" (UniqueName: \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\") pod \"904fa85f-1c68-4848-ad2b-fce99694c124\" (UID: \"904fa85f-1c68-4848-ad2b-fce99694c124\") "
Oct 30 22:19:16.588785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.588761    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca" pod "904fa85f-1c68-4848-ad2b-fce99694c124" (UID: "904fa85f-1c68-4848-ad2b-fce99694c124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:16.588923 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.588897    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:16.590662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.590554    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n" pod "904fa85f-1c68-4848-ad2b-fce99694c124" (UID: "904fa85f-1c68-4848-ad2b-fce99694c124"). InnerVolumeSpecName "kube-api-access-27s6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:16.591054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.591031    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca" pod "904fa85f-1c68-4848-ad2b-fce99694c124" (UID: "904fa85f-1c68-4848-ad2b-fce99694c124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:19:16.690384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.690340    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/904fa85f-1c68-4848-ad2b-fce99694c124-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:16.690384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:16.690375    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27s6n\" (UniqueName: \"kubernetes.io/projected/904fa85f-1c68-4848-ad2b-fce99694c124-kube-api-access-27s6n\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:17.315501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:17.315419    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"
Oct 30 22:19:17.315501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:17.315448    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw" event={"ID":"904fa85f-1c68-4848-ad2b-fce99694c124","Type":"ContainerDied","Data":"aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc"}
Oct 30 22:19:17.315501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:17.315499    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aad334b6b17fa7c54de0b61b0f3429cdf725a24ad47e47256c83c79cb12977cc"
Oct 30 22:19:20.808116 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:20.808048    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:19:26.358178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:26.358142    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:19:30.795182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.795146    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:19:30.795706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.795690    4882 cpu_manager.go:414] "containerMap: removing container" podUID="904fa85f-1c68-4848-ad2b-fce99694c124" containerName="revoke-keys"
Oct 30 22:19:30.795774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.795708    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="904fa85f-1c68-4848-ad2b-fce99694c124" containerName="revoke-keys"
Oct 30 22:19:30.795814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.795786    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="904fa85f-1c68-4848-ad2b-fce99694c124" containerName="revoke-keys"
Oct 30 22:19:30.799349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.799328    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.845152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.845110    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mhp6r\" (UniqueName: \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.845330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.845241    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.858999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.858966    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:19:30.905839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.905809    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-ca6d0b\""
Oct 30 22:19:30.906027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.905869    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-ca6d0b-dockercfg-42fgl\""
Oct 30 22:19:30.945960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.945925    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.946160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.946117    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mhp6r\" (UniqueName: \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.946710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.946690    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:30.958713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.958684    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:19:30.962636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.962614    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:30.990640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:30.990608    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:19:31.008161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.008129    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mhp6r\" (UniqueName: \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\") pod \"payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:31.046989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.046905    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ckxx\" (UniqueName: \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.046989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.046953    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.113505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.113473    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:31.148182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.148145    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ckxx\" (UniqueName: \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.148346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.148292    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.149001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.148980    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.210171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.210141    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ckxx\" (UniqueName: \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\") pod \"payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.276603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.276562    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:31.492704 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:31.492666    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc947cb_5986_412d_9399_d075c49b868f.slice/crio-9fbfc726069b04b12db1429d3e10e773733db83ddb8aa96e9380c91cb6b8fa3d WatchSource:0}: Error finding container 9fbfc726069b04b12db1429d3e10e773733db83ddb8aa96e9380c91cb6b8fa3d: Status 404 returned error can't find the container with id 9fbfc726069b04b12db1429d3e10e773733db83ddb8aa96e9380c91cb6b8fa3d
Oct 30 22:19:31.496047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.496022    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:19:31.660135 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:31.660098    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01760d91_50ba_4479_ab47_9f0750d6b3b0.slice/crio-b7e4e62fde24b7c6a91f7dcfac0bcfbbceab2020a984223b6a9b1b30cd0779b1 WatchSource:0}: Error finding container b7e4e62fde24b7c6a91f7dcfac0bcfbbceab2020a984223b6a9b1b30cd0779b1: Status 404 returned error can't find the container with id b7e4e62fde24b7c6a91f7dcfac0bcfbbceab2020a984223b6a9b1b30cd0779b1
Oct 30 22:19:31.663317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:31.663287    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:19:32.414535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.414492    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" event={"ID":"ecc947cb-5986-412d-9399-d075c49b868f","Type":"ContainerStarted","Data":"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"}
Oct 30 22:19:32.415055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.414622    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" event={"ID":"ecc947cb-5986-412d-9399-d075c49b868f","Type":"ContainerStarted","Data":"9fbfc726069b04b12db1429d3e10e773733db83ddb8aa96e9380c91cb6b8fa3d"}
Oct 30 22:19:32.415055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.414664    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:32.416342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.416313    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" event={"ID":"01760d91-50ba-4479-ab47-9f0750d6b3b0","Type":"ContainerStarted","Data":"981d89a917f060869123c3ce663103b75470947a5ce185f3d0e335e5cf2b2173"}
Oct 30 22:19:32.416466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.416366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" event={"ID":"01760d91-50ba-4479-ab47-9f0750d6b3b0","Type":"ContainerStarted","Data":"b7e4e62fde24b7c6a91f7dcfac0bcfbbceab2020a984223b6a9b1b30cd0779b1"}
Oct 30 22:19:32.416466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.416395    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:32.535919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.535858    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" podStartSLOduration=2.535844149 podStartE2EDuration="2.535844149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:32.533388344 +0000 UTC m=+2495.706712769" watchObservedRunningTime="2026-10-30 22:19:32.535844149 +0000 UTC m=+2495.709168621"
Oct 30 22:19:32.652045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:32.651993    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" podStartSLOduration=2.651977328 podStartE2EDuration="2.651977328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:32.650991265 +0000 UTC m=+2495.824315689" watchObservedRunningTime="2026-10-30 22:19:32.651977328 +0000 UTC m=+2495.825301754"
Oct 30 22:19:35.852415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.852380    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:19:35.856681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.856660    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:35.901427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.901389    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vjnzl\" (UniqueName: \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:35.901648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.901578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:35.911816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.911656    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:19:35.958417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.958375    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d2f1cf\""
Oct 30 22:19:35.958647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:35.958526    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d2f1cf-dockercfg-v7zb6\""
Oct 30 22:19:36.003043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.003000    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vjnzl\" (UniqueName: \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:36.003234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.003214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:36.003998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.003966    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:36.011470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.011443    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:19:36.015514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.015497    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.057215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.057185    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:19:36.064688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.064655    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vjnzl\" (UniqueName: \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:36.104948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.104856    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.105093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.105048    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28cq5\" (UniqueName: \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.170795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.170752    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:36.206525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.206489    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28cq5\" (UniqueName: \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.206708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.206580    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.207246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.207224    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.268446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.268402    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-28cq5\" (UniqueName: \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\") pod \"payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.329368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.329324    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:36.543181 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:36.543126    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc3a16fd5_b3a2_4d66_856d_4b2228c3ac24.slice/crio-48d442e8b8ed63cf42cbed6bbf3e149ad232e146a39afcd3478b8188f49b1736 WatchSource:0}: Error finding container 48d442e8b8ed63cf42cbed6bbf3e149ad232e146a39afcd3478b8188f49b1736: Status 404 returned error can't find the container with id 48d442e8b8ed63cf42cbed6bbf3e149ad232e146a39afcd3478b8188f49b1736
Oct 30 22:19:36.545961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.545937    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:19:36.706949 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:36.706913    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod84bfcb79_f423_460f_b6db_f46d715148fd.slice/crio-da20e4b02d5fd0688daad1763a6882a55acc717aa34fc19b100250b098d33333 WatchSource:0}: Error finding container da20e4b02d5fd0688daad1763a6882a55acc717aa34fc19b100250b098d33333: Status 404 returned error can't find the container with id da20e4b02d5fd0688daad1763a6882a55acc717aa34fc19b100250b098d33333
Oct 30 22:19:36.709773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:36.709750    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:19:37.446844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.446803    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" event={"ID":"84bfcb79-f423-460f-b6db-f46d715148fd","Type":"ContainerStarted","Data":"f5407857049713e0c2786b3128074cf6ba15885ce55a67f115650ce3980410fb"}
Oct 30 22:19:37.447288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.447000    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:37.447288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.447035    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" event={"ID":"84bfcb79-f423-460f-b6db-f46d715148fd","Type":"ContainerStarted","Data":"da20e4b02d5fd0688daad1763a6882a55acc717aa34fc19b100250b098d33333"}
Oct 30 22:19:37.448497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.448469    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" event={"ID":"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24","Type":"ContainerStarted","Data":"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"}
Oct 30 22:19:37.448618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.448514    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" event={"ID":"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24","Type":"ContainerStarted","Data":"48d442e8b8ed63cf42cbed6bbf3e149ad232e146a39afcd3478b8188f49b1736"}
Oct 30 22:19:37.448618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.448560    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:37.566090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.566034    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" podStartSLOduration=2.56601842 podStartE2EDuration="2.56601842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:37.56337205 +0000 UTC m=+2500.736696475" watchObservedRunningTime="2026-10-30 22:19:37.56601842 +0000 UTC m=+2500.739342896"
Oct 30 22:19:37.682449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:37.682386    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" podStartSLOduration=2.682365508 podStartE2EDuration="2.682365508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:37.680535799 +0000 UTC m=+2500.853860226" watchObservedRunningTime="2026-10-30 22:19:37.682365508 +0000 UTC m=+2500.855689935"
Oct 30 22:19:38.393051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.393005    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:19:38.396878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.396846    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.431751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.431716    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.431937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.431781    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6pdf\" (UniqueName: \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.453350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.453317    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f4a424\""
Oct 30 22:19:38.453873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.453472    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f4a424-dockercfg-4nq5j\""
Oct 30 22:19:38.456819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.456786    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:19:38.531644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.531611    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:19:38.533242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.533211    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.533377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.533281    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6pdf\" (UniqueName: \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.534018 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.533995    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.535768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.535752    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.592683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.592641    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:19:38.597194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.597160    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6pdf\" (UniqueName: \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\") pod \"payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.634789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.634753    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr5rl\" (UniqueName: \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.635000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.634884    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.711949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.711852    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:38.735738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.735692    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.735892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.735852    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr5rl\" (UniqueName: \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.736722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.736696    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.798909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.798865    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr5rl\" (UniqueName: \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\") pod \"payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.849751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.849716    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:38.898067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.898022    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:19:38.898340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:38.898311    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" podUID="08901306-e819-40a1-ad61-8329776fb5b8" containerName="payload-pre-processing" containerID="cri-o://055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095" gracePeriod=30
Oct 30 22:19:39.088856 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:39.088820    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4f19f7a_cad6_4be1_b6ff_bd124c8acadb.slice/crio-cdb0c90620fd85ac2940b825a0654e03256a9d26e47a7b5ed6ac3f9c2f968932 WatchSource:0}: Error finding container cdb0c90620fd85ac2940b825a0654e03256a9d26e47a7b5ed6ac3f9c2f968932: Status 404 returned error can't find the container with id cdb0c90620fd85ac2940b825a0654e03256a9d26e47a7b5ed6ac3f9c2f968932
Oct 30 22:19:39.090807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.090547    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:19:39.168369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.168348    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:19:39.233610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.233569    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:19:39.235755 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:39.235722    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55516d86_a1f5_4369_aaeb_5170f0e2ebb7.slice/crio-7b35835737bdbc71f15bfb4408fd55c2869dc9aa710ab4635345085f292822b6 WatchSource:0}: Error finding container 7b35835737bdbc71f15bfb4408fd55c2869dc9aa710ab4635345085f292822b6: Status 404 returned error can't find the container with id 7b35835737bdbc71f15bfb4408fd55c2869dc9aa710ab4635345085f292822b6
Oct 30 22:19:39.245211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.245191    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\" (UniqueName: \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\") pod \"08901306-e819-40a1-ad61-8329776fb5b8\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") "
Oct 30 22:19:39.245322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.245235    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\") pod \"08901306-e819-40a1-ad61-8329776fb5b8\" (UID: \"08901306-e819-40a1-ad61-8329776fb5b8\") "
Oct 30 22:19:39.245690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.245670    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume" pod "08901306-e819-40a1-ad61-8329776fb5b8" (UID: "08901306-e819-40a1-ad61-8329776fb5b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:19:39.247313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.247291    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh" pod "08901306-e819-40a1-ad61-8329776fb5b8" (UID: "08901306-e819-40a1-ad61-8329776fb5b8"). InnerVolumeSpecName "kube-api-access-xhxsh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:19:39.347285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.347254    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhxsh\" (UniqueName: \"kubernetes.io/projected/08901306-e819-40a1-ad61-8329776fb5b8-kube-api-access-xhxsh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:39.347285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.347286    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08901306-e819-40a1-ad61-8329776fb5b8-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:19:39.467181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.467135    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" event={"ID":"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb","Type":"ContainerStarted","Data":"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"}
Oct 30 22:19:39.467696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.467206    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" event={"ID":"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb","Type":"ContainerStarted","Data":"cdb0c90620fd85ac2940b825a0654e03256a9d26e47a7b5ed6ac3f9c2f968932"}
Oct 30 22:19:39.467696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.467399    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:39.468807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.468777    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" event={"ID":"55516d86-a1f5-4369-aaeb-5170f0e2ebb7","Type":"ContainerStarted","Data":"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"}
Oct 30 22:19:39.468938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.468831    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" event={"ID":"55516d86-a1f5-4369-aaeb-5170f0e2ebb7","Type":"ContainerStarted","Data":"7b35835737bdbc71f15bfb4408fd55c2869dc9aa710ab4635345085f292822b6"}
Oct 30 22:19:39.468938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.468864    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:39.470020 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.469994    4882 generic.go:356] "Generic (PLEG): container finished" podID="08901306-e819-40a1-ad61-8329776fb5b8" containerID="055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095" exitCode=0
Oct 30 22:19:39.470114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.470026    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"
Oct 30 22:19:39.470114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.470036    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" event={"ID":"08901306-e819-40a1-ad61-8329776fb5b8","Type":"ContainerDied","Data":"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"}
Oct 30 22:19:39.470114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.470067    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj" event={"ID":"08901306-e819-40a1-ad61-8329776fb5b8","Type":"ContainerDied","Data":"faaea3ad47bed8e4ce901005319dd078b6952df2cede78c7fe3b38f56b4cc697"}
Oct 30 22:19:39.470114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.470089    4882 scope.go:122] "RemoveContainer" containerID="055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"
Oct 30 22:19:39.485459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.485436    4882 scope.go:122] "RemoveContainer" containerID="055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"
Oct 30 22:19:39.485960 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:39.485934    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095\": container with ID starting with 055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095 not found: ID does not exist" containerID="055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"
Oct 30 22:19:39.486043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.485968    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095"} err="failed to get container status \"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095\": rpc error: code = NotFound desc = could not find container \"055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095\": container with ID starting with 055b288b77036f85612fabeb9e5f95882db9fd812365c7b8f9b7bb93d08cc095 not found: ID does not exist"
Oct 30 22:19:39.585957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.585879    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" podStartSLOduration=1.585860713 podStartE2EDuration="1.585860713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:39.583841425 +0000 UTC m=+2502.757165861" watchObservedRunningTime="2026-10-30 22:19:39.585860713 +0000 UTC m=+2502.759185134"
Oct 30 22:19:39.696896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.696858    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:19:39.701705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.701676    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-fe745d-b94fb6ff9-pb9cj"]
Oct 30 22:19:39.819231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:39.819172    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" podStartSLOduration=1.819156394 podStartE2EDuration="1.819156394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:39.81718144 +0000 UTC m=+2502.990505867" watchObservedRunningTime="2026-10-30 22:19:39.819156394 +0000 UTC m=+2502.992480818"
Oct 30 22:19:40.568729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:40.568692    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"]
Oct 30 22:19:40.571160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:40.571138    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-fe745d-5cczw"]
Oct 30 22:19:40.943392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:40.943355    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08901306-e819-40a1-ad61-8329776fb5b8" path="/var/lib/kubelet/pods/08901306-e819-40a1-ad61-8329776fb5b8/volumes"
Oct 30 22:19:40.943809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:40.943796    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="904fa85f-1c68-4848-ad2b-fce99694c124" path="/var/lib/kubelet/pods/904fa85f-1c68-4848-ad2b-fce99694c124/volumes"
Oct 30 22:19:43.425268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:43.425238    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:19:43.425777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:43.425309    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:19:48.461670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:48.461638    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:19:48.462109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:48.461825    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:19:50.490237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:50.490206    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:19:50.490645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:50.490284    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:19:50.810927 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:19:50.810832    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:19:54.894650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:54.894612    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:19:58.369754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.369718    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:19:58.370508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.370481    4882 cpu_manager.go:414] "containerMap: removing container" podUID="08901306-e819-40a1-ad61-8329776fb5b8" containerName="payload-pre-processing"
Oct 30 22:19:58.370663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.370508    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08901306-e819-40a1-ad61-8329776fb5b8" containerName="payload-pre-processing"
Oct 30 22:19:58.370737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.370708    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="08901306-e819-40a1-ad61-8329776fb5b8" containerName="payload-pre-processing"
Oct 30 22:19:58.374044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.374023    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.428366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.428331    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:19:58.456563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.456531    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.456771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.456622    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkcvf\" (UniqueName: \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.475158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.475126    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-a-f7c488\""
Oct 30 22:19:58.475158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.475135    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-a-f7c488-dockercfg-x7j94\""
Oct 30 22:19:58.557676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.557632    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkcvf\" (UniqueName: \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.557924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.557795    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.558551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.558523    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.619438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.619405    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkcvf\" (UniqueName: \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\") pod \"payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") " pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:58.688648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:58.688535    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:59.076807 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:19:59.076771    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fbfd9b6_462b_4427_9d1f_a76f337ef956.slice/crio-fdbadd301502744798f15fc0f51c9e939a464a0f6898dedf8b61d30890f169ce WatchSource:0}: Error finding container fdbadd301502744798f15fc0f51c9e939a464a0f6898dedf8b61d30890f169ce: Status 404 returned error can't find the container with id fdbadd301502744798f15fc0f51c9e939a464a0f6898dedf8b61d30890f169ce
Oct 30 22:19:59.078914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:59.078884    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:19:59.620068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:59.620034    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" event={"ID":"6fbfd9b6-462b-4427-9d1f-a76f337ef956","Type":"ContainerStarted","Data":"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"}
Oct 30 22:19:59.620479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:59.620081    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" event={"ID":"6fbfd9b6-462b-4427-9d1f-a76f337ef956","Type":"ContainerStarted","Data":"fdbadd301502744798f15fc0f51c9e939a464a0f6898dedf8b61d30890f169ce"}
Oct 30 22:19:59.620479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:59.620185    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:19:59.739379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:19:59.739318    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" podStartSLOduration=1.7393035700000001 podStartE2EDuration="1.73930357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:19:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:19:59.737174192 +0000 UTC m=+2522.910498611" watchObservedRunningTime="2026-10-30 22:19:59.73930357 +0000 UTC m=+2522.912627995"
Oct 30 22:20:00.595783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.595749    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:20:00.599801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.599775    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.653170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.653130    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-63a282\""
Oct 30 22:20:00.653575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.653323    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-63a282-dockercfg-kk79c\""
Oct 30 22:20:00.682890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.682850    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbdl2\" (UniqueName: \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.683071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.683002    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.706381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.706348    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:20:00.784533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.784494    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbdl2\" (UniqueName: \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.784780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.784622    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.785351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.785326    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.846157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.846077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbdl2\" (UniqueName: \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\") pod \"payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:00.914604 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:00.914030    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:01.296198 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:20:01.296169    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6179c80_1db2_4877_9abe_9f6aaffe97f1.slice/crio-be71a2f658319c9228bd27e9279902eb4d429ce9c4aa68745ff719c03c59d087 WatchSource:0}: Error finding container be71a2f658319c9228bd27e9279902eb4d429ce9c4aa68745ff719c03c59d087: Status 404 returned error can't find the container with id be71a2f658319c9228bd27e9279902eb4d429ce9c4aa68745ff719c03c59d087
Oct 30 22:20:01.299016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.298993    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:20:01.394399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.394365    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:20:01.398382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.398362    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.454405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.454370    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:20:01.494057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.494024    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.494218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.494090    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg84q\" (UniqueName: \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.594991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.594960    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.595185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.595018    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg84q\" (UniqueName: \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.595719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.595698    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.636956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.636915    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" event={"ID":"c6179c80-1db2-4877-9abe-9f6aaffe97f1","Type":"ContainerStarted","Data":"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"}
Oct 30 22:20:01.637143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.636969    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" event={"ID":"c6179c80-1db2-4877-9abe-9f6aaffe97f1","Type":"ContainerStarted","Data":"be71a2f658319c9228bd27e9279902eb4d429ce9c4aa68745ff719c03c59d087"}
Oct 30 22:20:01.637143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.637047    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:01.656813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.656780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg84q\" (UniqueName: \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\") pod \"payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.712272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.712241    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:01.759817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:01.759761    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" podStartSLOduration=1.759744563 podStartE2EDuration="1.759744563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:01.757793106 +0000 UTC m=+2524.931117533" watchObservedRunningTime="2026-10-30 22:20:01.759744563 +0000 UTC m=+2524.933068988"
Oct 30 22:20:02.089301 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:20:02.089265    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36be8729_107f_4459_b3f2_f4f06d6a90a9.slice/crio-903a41a6e77ba884f385e1e43e3f3b7b0e724523aa7cc9588d358e3259700dab WatchSource:0}: Error finding container 903a41a6e77ba884f385e1e43e3f3b7b0e724523aa7cc9588d358e3259700dab: Status 404 returned error can't find the container with id 903a41a6e77ba884f385e1e43e3f3b7b0e724523aa7cc9588d358e3259700dab
Oct 30 22:20:02.092560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:02.092535    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:20:02.644909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:02.644864    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" event={"ID":"36be8729-107f-4459-b3f2-f4f06d6a90a9","Type":"ContainerStarted","Data":"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"}
Oct 30 22:20:02.645153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:02.644930    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" event={"ID":"36be8729-107f-4459-b3f2-f4f06d6a90a9","Type":"ContainerStarted","Data":"903a41a6e77ba884f385e1e43e3f3b7b0e724523aa7cc9588d358e3259700dab"}
Oct 30 22:20:02.645355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:02.645088    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:02.764770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:02.764719    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" podStartSLOduration=1.764704761 podStartE2EDuration="1.764704761s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:20:02.763707906 +0000 UTC m=+2525.937032332" watchObservedRunningTime="2026-10-30 22:20:02.764704761 +0000 UTC m=+2525.938029192"
Oct 30 22:20:10.629046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:10.628966    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:20:12.647505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:12.647472    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:20:13.654762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:13.654726    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:20:20.808889 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:20:20.808839    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:20:39.854341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:39.854307    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:20:50.813787 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:20:50.813742    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:20:58.790145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:20:58.790110    4882 scope.go:122] "RemoveContainer" containerID="5788e7a1515a9504ea3680a99f07456f653fe81a47dbc1ae79192f3203e31df5"
Oct 30 22:21:05.205228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.205184    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:21:05.210641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.210616    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.264121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.264082    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:21:05.306105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.306059    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.306293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.306236    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wprxs\" (UniqueName: \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.311208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.311177    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-85d00c\""
Oct 30 22:21:05.311357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.311178    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-85d00c-dockercfg-z59h7\""
Oct 30 22:21:05.348228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.348186    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:21:05.352330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.352307    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.406764 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.406727    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:21:05.407048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.407020    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjxrb\" (UniqueName: \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.407135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.407070    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.407229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.407202    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.407365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.407263    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wprxs\" (UniqueName: \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.408040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.408011    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.469455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.469379    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wprxs\" (UniqueName: \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\") pod \"payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.508385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.508339    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjxrb\" (UniqueName: \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.508634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.508446    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.509203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.509183    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.524159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.524124    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:05.570814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.570782    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjxrb\" (UniqueName: \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\") pod \"payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.666894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.666855    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:05.899458 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:21:05.899416    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9c7c856_0801_4bf1_a454_07adbdd6736e.slice/crio-cd8f65e34c45f2e1b3dfee1966e09b67a1103d0b93d674a873dec5fc3a1ad7fc WatchSource:0}: Error finding container cd8f65e34c45f2e1b3dfee1966e09b67a1103d0b93d674a873dec5fc3a1ad7fc: Status 404 returned error can't find the container with id cd8f65e34c45f2e1b3dfee1966e09b67a1103d0b93d674a873dec5fc3a1ad7fc
Oct 30 22:21:05.901210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:05.901187    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:21:06.043533 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:21:06.043491    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb57a61d_6086_4d2c_8d18_66eaec59d0bd.slice/crio-4231725c1ab96aeff21d1eec3319d7db582c75121c5eb4f4f8d1f2859af15f4e WatchSource:0}: Error finding container 4231725c1ab96aeff21d1eec3319d7db582c75121c5eb4f4f8d1f2859af15f4e: Status 404 returned error can't find the container with id 4231725c1ab96aeff21d1eec3319d7db582c75121c5eb4f4f8d1f2859af15f4e
Oct 30 22:21:06.045220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.045200    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:21:06.059741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.059711    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" event={"ID":"d9c7c856-0801-4bf1-a454-07adbdd6736e","Type":"ContainerStarted","Data":"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"}
Oct 30 22:21:06.059904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.059771    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" event={"ID":"d9c7c856-0801-4bf1-a454-07adbdd6736e","Type":"ContainerStarted","Data":"cd8f65e34c45f2e1b3dfee1966e09b67a1103d0b93d674a873dec5fc3a1ad7fc"}
Oct 30 22:21:06.059904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.059806    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:06.061026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.061000    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" event={"ID":"eb57a61d-6086-4d2c-8d18-66eaec59d0bd","Type":"ContainerStarted","Data":"4231725c1ab96aeff21d1eec3319d7db582c75121c5eb4f4f8d1f2859af15f4e"}
Oct 30 22:21:06.177618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:06.177553    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" podStartSLOduration=1.177538394 podStartE2EDuration="1.177538394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:06.176998276 +0000 UTC m=+2589.350322700" watchObservedRunningTime="2026-10-30 22:21:06.177538394 +0000 UTC m=+2589.350862821"
Oct 30 22:21:07.073322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:07.073278    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" event={"ID":"eb57a61d-6086-4d2c-8d18-66eaec59d0bd","Type":"ContainerStarted","Data":"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"}
Oct 30 22:21:07.073900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:07.073873    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:07.192691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:07.192633    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" podStartSLOduration=2.192616409 podStartE2EDuration="2.192616409s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:07.18997244 +0000 UTC m=+2590.363296868" watchObservedRunningTime="2026-10-30 22:21:07.192616409 +0000 UTC m=+2590.365940837"
Oct 30 22:21:11.805644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:11.805603    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:21:17.076904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:17.076867    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:21:19.088888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:19.088860    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:21:20.810248 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:21:20.810212    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:21:21.370515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.370477    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:21:21.374504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.374483    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.462295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.462247    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:21:21.500264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500223    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500424    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szj6j\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500578    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500685    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500739    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500777    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.500892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.500876    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.508391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.508367    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-a-f7c488-openshift-default-dockercfg-2kvn7\""
Oct 30 22:21:21.602428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602393    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602436    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602629    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602665    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szj6j\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602705    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602751    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.602830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602796    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602834    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602839    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602881    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.602909    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.603168    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.603266    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.603548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.603526    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.605053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.605020    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.605264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.605246    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.670755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.670673    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.672058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.672034    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-szj6j\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\") pod \"e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") " pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:21.688971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:21.688938    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:22.114573 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:21:22.114543    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4492fb74_46d3_488b_8ebe_229689150d0c.slice/crio-8a586faada31e21c659d19bb3f737c7bb97576724f0222498c48ca4a5154b64c WatchSource:0}: Error finding container 8a586faada31e21c659d19bb3f737c7bb97576724f0222498c48ca4a5154b64c: Status 404 returned error can't find the container with id 8a586faada31e21c659d19bb3f737c7bb97576724f0222498c48ca4a5154b64c
Oct 30 22:21:22.118274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:22.118243    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:21:22.118467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:22.118416    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:21:22.118633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:22.118576    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:21:22.118712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:22.118700    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:21:22.166316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:22.166270    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" event={"ID":"4492fb74-46d3-488b-8ebe-229689150d0c","Type":"ContainerStarted","Data":"8a586faada31e21c659d19bb3f737c7bb97576724f0222498c48ca4a5154b64c"}
Oct 30 22:21:23.174271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:23.174232    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" event={"ID":"4492fb74-46d3-488b-8ebe-229689150d0c","Type":"ContainerStarted","Data":"cc5651fb456f01bd353865e54887a300ab417eb210cd9ddad58662011eeced6a"}
Oct 30 22:21:23.364491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:23.364437    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podStartSLOduration=2.364423631 podStartE2EDuration="2.364423631s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:23.313408893 +0000 UTC m=+2606.486733320" watchObservedRunningTime="2026-10-30 22:21:23.364423631 +0000 UTC m=+2606.537748057"
Oct 30 22:21:23.689680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:23.689645    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:23.691527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:23.691503    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:23.691698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:23.691563    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:24.689700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:24.689665    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:24.690126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:24.689721    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:25.689814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:25.689766    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:25.690239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:25.689861    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:26.689811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:26.689777    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:26.690016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:26.689836    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:27.689575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:27.689537    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:27.689811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:27.689628    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:28.689505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:28.689464    4882 patch_prober.go:29] interesting pod/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused" start-of-body=
Oct 30 22:21:28.689970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:28.689521    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.207:15021/healthz/ready\": dial tcp 10.132.0.207:15021: connect: connection refused"
Oct 30 22:21:29.694235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:29.694204    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:29.694690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:29.694552    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:29.695425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:29.695402    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:21:36.410231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.410188    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:21:36.414803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.414782    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.472673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.472640    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:21:36.488763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.488726    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.488971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.488790    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtv8c\" (UniqueName: \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.518945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.518904    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-fc3918\""
Oct 30 22:21:36.518945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.518904    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-fc3918-dockercfg-z62ch\""
Oct 30 22:21:36.589758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.589717    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.589966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.589771    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtv8c\" (UniqueName: \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.590571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.590544    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.652720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.652681    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtv8c\" (UniqueName: \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\") pod \"payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:36.729844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:36.729761    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:37.108944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.108923    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:21:37.111526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.111481    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:21:37.224682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.224648    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:21:37.266072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.266025    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" event={"ID":"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b","Type":"ContainerStarted","Data":"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"}
Oct 30 22:21:37.266292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.266082    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" event={"ID":"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b","Type":"ContainerStarted","Data":"1fba8210c84649f05978bbcd42a6842b282a5453486f867143ed773d688cfe35"}
Oct 30 22:21:37.266292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.266119    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:37.468354 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:37.468239    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" podStartSLOduration=1.468223202 podStartE2EDuration="1.468223202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:37.465027772 +0000 UTC m=+2620.638352198" watchObservedRunningTime="2026-10-30 22:21:37.468223202 +0000 UTC m=+2620.641547629"
Oct 30 22:21:40.217812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.217777    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:21:40.222275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.222248    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.276841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.276805    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-63a282\"/\"kube-root-ca.crt\""
Oct 30 22:21:40.278869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.278836    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:21:40.326167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.326134    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-63a282\"/\"default-dockercfg-llxt7\""
Oct 30 22:21:40.327491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.327472    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-63a282\"/\"ipp-route-63a282-kserve-self-signed-certs\""
Oct 30 22:21:40.336064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336032    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.336275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336249    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2z8g6\" (UniqueName: \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.336318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336302    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.336394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336383    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.336431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336403    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.336493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.336477    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.376289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.376260    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-63a282\"/\"openshift-service-ca.crt\""
Oct 30 22:21:40.438030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.437994    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2z8g6\" (UniqueName: \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438042    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438067    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438126    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438149    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438531    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438546    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.438679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.438640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.440573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.440543    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.440740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.440720    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.502988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.502905    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2z8g6\" (UniqueName: \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\") pod \"ipp-route-63a282-kserve-5f5f5d99cc-vx68m\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") " pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.535989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.535951    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:40.924018 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:21:40.923985    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5bf34c12_6fe9_42c0_8f30_b669af3d94c8.slice/crio-b9ae086cc783ada69f4f3b765c99b57fa6981225a0517f6d42ddb701a7356885 WatchSource:0}: Error finding container b9ae086cc783ada69f4f3b765c99b57fa6981225a0517f6d42ddb701a7356885: Status 404 returned error can't find the container with id b9ae086cc783ada69f4f3b765c99b57fa6981225a0517f6d42ddb701a7356885
Oct 30 22:21:40.924167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:40.924133    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:21:41.298247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:41.298142    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" event={"ID":"5bf34c12-6fe9-42c0-8f30-b669af3d94c8","Type":"ContainerStarted","Data":"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"}
Oct 30 22:21:41.298247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:41.298195    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" event={"ID":"5bf34c12-6fe9-42c0-8f30-b669af3d94c8","Type":"ContainerStarted","Data":"b9ae086cc783ada69f4f3b765c99b57fa6981225a0517f6d42ddb701a7356885"}
Oct 30 22:21:41.420090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:41.420032    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" podStartSLOduration=1.420016928 podStartE2EDuration="1.420016928s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:41.417003339 +0000 UTC m=+2624.590327769" watchObservedRunningTime="2026-10-30 22:21:41.420016928 +0000 UTC m=+2624.593341355"
Oct 30 22:21:41.536954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:41.536915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:42.109546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.109498    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" podUID="84bfcb79-f423-460f-b6db-f46d715148fd" containerName="payload-processing" containerID="cri-o://f5407857049713e0c2786b3128074cf6ba15885ce55a67f115650ce3980410fb" gracePeriod=25
Oct 30 22:21:42.225902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.225866    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" podUID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" containerName="payload-pre-processing" containerID="cri-o://475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234" gracePeriod=25
Oct 30 22:21:42.308392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.308350    4882 generic.go:356] "Generic (PLEG): container finished" podID="84bfcb79-f423-460f-b6db-f46d715148fd" containerID="f5407857049713e0c2786b3128074cf6ba15885ce55a67f115650ce3980410fb" exitCode=0
Oct 30 22:21:42.308912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.308416    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" event={"ID":"84bfcb79-f423-460f-b6db-f46d715148fd","Type":"ContainerDied","Data":"f5407857049713e0c2786b3128074cf6ba15885ce55a67f115650ce3980410fb"}
Oct 30 22:21:42.410040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.410001    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:21:42.461231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.461186    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\") pod \"84bfcb79-f423-460f-b6db-f46d715148fd\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") "
Oct 30 22:21:42.461418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.461322    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\" (UniqueName: \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\") pod \"84bfcb79-f423-460f-b6db-f46d715148fd\" (UID: \"84bfcb79-f423-460f-b6db-f46d715148fd\") "
Oct 30 22:21:42.461703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.461675    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume" pod "84bfcb79-f423-460f-b6db-f46d715148fd" (UID: "84bfcb79-f423-460f-b6db-f46d715148fd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:42.462260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.462234    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/84bfcb79-f423-460f-b6db-f46d715148fd-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.465605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.465566    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5" pod "84bfcb79-f423-460f-b6db-f46d715148fd" (UID: "84bfcb79-f423-460f-b6db-f46d715148fd"). InnerVolumeSpecName "kube-api-access-28cq5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:42.546636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.546599    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:21:42.563669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.563634    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28cq5\" (UniqueName: \"kubernetes.io/projected/84bfcb79-f423-460f-b6db-f46d715148fd-kube-api-access-28cq5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.664243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.664154    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\") pod \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") "
Oct 30 22:21:42.664476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.664314    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\" (UniqueName: \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\") pod \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\" (UID: \"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24\") "
Oct 30 22:21:42.664676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.664529    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume" pod "c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" (UID: "c3a16fd5-b3a2-4d66-856d-4b2228c3ac24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:21:42.665045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.665024    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:42.667456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.667419    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl" pod "c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" (UID: "c3a16fd5-b3a2-4d66-856d-4b2228c3ac24"). InnerVolumeSpecName "kube-api-access-vjnzl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:21:42.765830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:42.765791    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vjnzl\" (UniqueName: \"kubernetes.io/projected/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24-kube-api-access-vjnzl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:21:43.316916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.316827    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"
Oct 30 22:21:43.316916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.316858    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm" event={"ID":"84bfcb79-f423-460f-b6db-f46d715148fd","Type":"ContainerDied","Data":"da20e4b02d5fd0688daad1763a6882a55acc717aa34fc19b100250b098d33333"}
Oct 30 22:21:43.317435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.316923    4882 scope.go:122] "RemoveContainer" containerID="f5407857049713e0c2786b3128074cf6ba15885ce55a67f115650ce3980410fb"
Oct 30 22:21:43.318414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.318380    4882 generic.go:356] "Generic (PLEG): container finished" podID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" containerID="475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234" exitCode=0
Oct 30 22:21:43.318496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.318443    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"
Oct 30 22:21:43.318496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.318455    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" event={"ID":"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24","Type":"ContainerDied","Data":"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"}
Oct 30 22:21:43.318668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.318523    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n" event={"ID":"c3a16fd5-b3a2-4d66-856d-4b2228c3ac24","Type":"ContainerDied","Data":"48d442e8b8ed63cf42cbed6bbf3e149ad232e146a39afcd3478b8188f49b1736"}
Oct 30 22:21:43.328835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.328812    4882 scope.go:122] "RemoveContainer" containerID="475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"
Oct 30 22:21:43.339011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.338980    4882 scope.go:122] "RemoveContainer" containerID="475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"
Oct 30 22:21:43.339401 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:21:43.339377    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234\": container with ID starting with 475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234 not found: ID does not exist" containerID="475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"
Oct 30 22:21:43.339485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.339410    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234"} err="failed to get container status \"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234\": rpc error: code = NotFound desc = could not find container \"475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234\": container with ID starting with 475d48d52a031a7dbf8dc9a72d86b2fb7e89735cb31a12e262a4ed726cc30234 not found: ID does not exist"
Oct 30 22:21:43.484422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.484380    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:21:43.534579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.534490    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-9f4586f9-spt2n"]
Oct 30 22:21:43.604707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.604665    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:21:43.606727 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:43.606699    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-699959cbc9-6lglm"]
Oct 30 22:21:44.957269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:44.946707    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84bfcb79-f423-460f-b6db-f46d715148fd" path="/var/lib/kubelet/pods/84bfcb79-f423-460f-b6db-f46d715148fd/volumes"
Oct 30 22:21:44.957269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:44.947411    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" path="/var/lib/kubelet/pods/c3a16fd5-b3a2-4d66-856d-4b2228c3ac24/volumes"
Oct 30 22:21:48.277164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:48.277133    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:21:50.536656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:50.536616    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:50.549365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:50.549336    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:50.808664 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:21:50.808544    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:21:51.388620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.388573    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:21:51.576633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.576579    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:21:51.577218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577203    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" containerName="payload-pre-processing"
Oct 30 22:21:51.577264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577219    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" containerName="payload-pre-processing"
Oct 30 22:21:51.577264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577248    4882 cpu_manager.go:414] "containerMap: removing container" podUID="84bfcb79-f423-460f-b6db-f46d715148fd" containerName="payload-processing"
Oct 30 22:21:51.577264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577253    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84bfcb79-f423-460f-b6db-f46d715148fd" containerName="payload-processing"
Oct 30 22:21:51.577369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577356    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c3a16fd5-b3a2-4d66-856d-4b2228c3ac24" containerName="payload-pre-processing"
Oct 30 22:21:51.577403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.577369    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="84bfcb79-f423-460f-b6db-f46d715148fd" containerName="payload-processing"
Oct 30 22:21:51.582342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.582318    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.635060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.635021    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:21:51.636400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.636374    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-93131d\""
Oct 30 22:21:51.636619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.636522    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-93131d-dockercfg-m7b9b\""
Oct 30 22:21:51.671695 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.671605    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.671846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.671758    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rx5kw\" (UniqueName: \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.772877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.772830    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.772877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.772881    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rx5kw\" (UniqueName: \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.773705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.773646    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.837130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.837094    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rx5kw\" (UniqueName: \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\") pod \"payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:51.896664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:51.896627    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:52.284224 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:21:52.284193    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab56c39e_ec82_4409_a289_fe89b6727b5e.slice/crio-82d5cfe8a138950dc8c23d1d6a215fc67ec47061ae0eea519423524e786ddb6b WatchSource:0}: Error finding container 82d5cfe8a138950dc8c23d1d6a215fc67ec47061ae0eea519423524e786ddb6b: Status 404 returned error can't find the container with id 82d5cfe8a138950dc8c23d1d6a215fc67ec47061ae0eea519423524e786ddb6b
Oct 30 22:21:52.286461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:52.286433    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:21:52.385931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:52.385897    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" event={"ID":"ab56c39e-ec82-4409-a289-fe89b6727b5e","Type":"ContainerStarted","Data":"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"}
Oct 30 22:21:52.386105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:52.385960    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" event={"ID":"ab56c39e-ec82-4409-a289-fe89b6727b5e","Type":"ContainerStarted","Data":"82d5cfe8a138950dc8c23d1d6a215fc67ec47061ae0eea519423524e786ddb6b"}
Oct 30 22:21:52.386105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:52.385990    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:21:52.504488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:52.504439    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" podStartSLOduration=1.504426036 podStartE2EDuration="1.504426036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:21:52.503141627 +0000 UTC m=+2635.676466055" watchObservedRunningTime="2026-10-30 22:21:52.504426036 +0000 UTC m=+2635.677750487"
Oct 30 22:21:56.919457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:21:56.919416    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:22:03.397319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:03.397289    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:22:20.809031 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:22:20.808994    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:22:21.621214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.621177    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:22:21.625346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.625324    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.682271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.682238    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:22:21.715434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.715397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.715683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.715485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qf5js\" (UniqueName: \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.729376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.729347    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-praxis-b-7fd427\""
Oct 30 22:22:21.729376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.729347    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-praxis-b-7fd427-dockercfg-kjrkh\""
Oct 30 22:22:21.782836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.782768    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:22:21.787001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.786973    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:21.816607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.816557    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.817108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.816679    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qf5js\" (UniqueName: \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.817800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.817776    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.840333 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.840291    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:22:21.881095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.881019    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qf5js\" (UniqueName: \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\") pod \"payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") " pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:21.917904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.917849    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8tg5\" (UniqueName: \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:21.918130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.918092    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:21.939558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:21.939519    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:22.018867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.018832    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8tg5\" (UniqueName: \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.019064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.018977    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.019958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.019933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.081264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.081221    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8tg5\" (UniqueName: \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\") pod \"payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") " pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.100760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.100729    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.314642 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:22:22.314577    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3501ece_3569_41f4_811b_b99e61ab5c3e.slice/crio-47c0476873637adc3de03c18eead9c88dbdefb9763768b6d5f6f2ffb59e022e5 WatchSource:0}: Error finding container 47c0476873637adc3de03c18eead9c88dbdefb9763768b6d5f6f2ffb59e022e5: Status 404 returned error can't find the container with id 47c0476873637adc3de03c18eead9c88dbdefb9763768b6d5f6f2ffb59e022e5
Oct 30 22:22:22.317287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.317258    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:22:22.475804 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:22:22.475769    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66812030_f241_474f_a2ae_49f75d195b20.slice/crio-dc1db4964af2f5190d1ee64343a8cb955c98d1f3fd990925212c8cbd409b8792 WatchSource:0}: Error finding container dc1db4964af2f5190d1ee64343a8cb955c98d1f3fd990925212c8cbd409b8792: Status 404 returned error can't find the container with id dc1db4964af2f5190d1ee64343a8cb955c98d1f3fd990925212c8cbd409b8792
Oct 30 22:22:22.478553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.478529    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:22:22.578294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.578258    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" event={"ID":"a3501ece-3569-41f4-811b-b99e61ab5c3e","Type":"ContainerStarted","Data":"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"}
Oct 30 22:22:22.578513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.578323    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:22.578513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.578353    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" event={"ID":"a3501ece-3569-41f4-811b-b99e61ab5c3e","Type":"ContainerStarted","Data":"47c0476873637adc3de03c18eead9c88dbdefb9763768b6d5f6f2ffb59e022e5"}
Oct 30 22:22:22.580169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.580140    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" event={"ID":"66812030-f241-474f-a2ae-49f75d195b20","Type":"ContainerStarted","Data":"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"}
Oct 30 22:22:22.580304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.580188    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" event={"ID":"66812030-f241-474f-a2ae-49f75d195b20","Type":"ContainerStarted","Data":"dc1db4964af2f5190d1ee64343a8cb955c98d1f3fd990925212c8cbd409b8792"}
Oct 30 22:22:22.580304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.580210    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:22.699011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.698954    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" podStartSLOduration=1.698938533 podStartE2EDuration="1.698938533s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:22.69532615 +0000 UTC m=+2665.868650578" watchObservedRunningTime="2026-10-30 22:22:22.698938533 +0000 UTC m=+2665.872262959"
Oct 30 22:22:22.813835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:22.813718    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" podStartSLOduration=1.8137045120000002 podStartE2EDuration="1.813704512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:22.812013446 +0000 UTC m=+2665.985337874" watchObservedRunningTime="2026-10-30 22:22:22.813704512 +0000 UTC m=+2665.987029032"
Oct 30 22:22:30.210034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.209996    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:22:30.214056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.214034    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.312220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.312176    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm4x7\" (UniqueName: \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.312411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.312367    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.317711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.317686    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-d2f1cf\""
Oct 30 22:22:30.317859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.317730    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-d2f1cf-dockercfg-4jfrb\""
Oct 30 22:22:30.320854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.320832    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:22:30.413010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.412972    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.413205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.413061    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm4x7\" (UniqueName: \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.413708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.413689    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.475247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.475171    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm4x7\" (UniqueName: \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\") pod \"payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.530048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.530007    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:30.904261 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:22:30.904224    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf14d2645_8c0b_4c92_8cda_5d5aba187bc7.slice/crio-359785f94ac753711e4df53ca5e7631829ae1a7e13ecee2c1894fd21be9319ed WatchSource:0}: Error finding container 359785f94ac753711e4df53ca5e7631829ae1a7e13ecee2c1894fd21be9319ed: Status 404 returned error can't find the container with id 359785f94ac753711e4df53ca5e7631829ae1a7e13ecee2c1894fd21be9319ed
Oct 30 22:22:30.906448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:30.906424    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:22:31.026235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.026203    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:22:31.030621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.030597    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.085482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.085443    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:22:31.119004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.118970    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnm6r\" (UniqueName: \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.119178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.119130    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.220226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.220134    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.220755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.220266    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnm6r\" (UniqueName: \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.220955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.220933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.282284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.282247    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnm6r\" (UniqueName: \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\") pod \"payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.345513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.345473    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:31.640868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.640832    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" event={"ID":"f14d2645-8c0b-4c92-8cda-5d5aba187bc7","Type":"ContainerStarted","Data":"95f7ef9b127b305f773280f70dc2c1ee9d3df9dc1fd94a2f36ab2acdb295e574"}
Oct 30 22:22:31.641078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.640892    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" event={"ID":"f14d2645-8c0b-4c92-8cda-5d5aba187bc7","Type":"ContainerStarted","Data":"359785f94ac753711e4df53ca5e7631829ae1a7e13ecee2c1894fd21be9319ed"}
Oct 30 22:22:31.641078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.640915    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:31.723308 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:22:31.723273    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23c02225_8584_47fe_8e58_f7ff75b8aa26.slice/crio-80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e WatchSource:0}: Error finding container 80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e: Status 404 returned error can't find the container with id 80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e
Oct 30 22:22:31.725180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.725039    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:22:31.762327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:31.762271    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" podStartSLOduration=1.762255937 podStartE2EDuration="1.762255937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:31.758780037 +0000 UTC m=+2674.932104465" watchObservedRunningTime="2026-10-30 22:22:31.762255937 +0000 UTC m=+2674.935580377"
Oct 30 22:22:32.649654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:32.649615    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" event={"ID":"23c02225-8584-47fe-8e58-f7ff75b8aa26","Type":"ContainerStarted","Data":"5c3f509a5814403b5feef10016a601e68ea98d8f78e17b5672a814ea60228635"}
Oct 30 22:22:32.650158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:32.649677    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" event={"ID":"23c02225-8584-47fe-8e58-f7ff75b8aa26","Type":"ContainerStarted","Data":"80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e"}
Oct 30 22:22:32.773073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:32.773016    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" podStartSLOduration=2.773000899 podStartE2EDuration="2.773000899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:32.771661429 +0000 UTC m=+2675.944985856" watchObservedRunningTime="2026-10-30 22:22:32.773000899 +0000 UTC m=+2675.946325326"
Oct 30 22:22:33.590031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:33.590001    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:22:33.590250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:33.590193    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:22:33.657661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:33.657631    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:36.593719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.593686    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:22:36.598461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.598439    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.685525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.685486    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.685805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.685780    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.685913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.685818    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bgs4\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.685969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.685947    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.686030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.686005    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.686554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.686097    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.686554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.686159    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.686554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.686194    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.686554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.686402    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.706353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.706322    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:22:36.724870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.724830    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-praxis-b-85d00c-openshift-default-dockercfg-tsjwc\""
Oct 30 22:22:36.787762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.787729    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.787762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.787768    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.787786    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bgs4\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.787911    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.787959    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788039    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788154    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788190    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788278    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788410    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788496    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788660    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788660    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.788875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.788828    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.790413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.790387    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.790780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.790740    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.849446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.849355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.849668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.849534    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bgs4\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\") pod \"e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") " pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:36.913415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:36.913378    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:37.342108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.342071    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:22:37.371345 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:22:37.371314    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a06f34_f671_49f3_961c_45adef24efac.slice/crio-6eae4235a3604a20ab6965a28546587458308492a7c8eba726c1f1e3862359ff WatchSource:0}: Error finding container 6eae4235a3604a20ab6965a28546587458308492a7c8eba726c1f1e3862359ff: Status 404 returned error can't find the container with id 6eae4235a3604a20ab6965a28546587458308492a7c8eba726c1f1e3862359ff
Oct 30 22:22:37.373933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.373896    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:22:37.374040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.373970    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:22:37.374040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.373999    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:22:37.689274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.689178    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" event={"ID":"69a06f34-f671-49f3-961c-45adef24efac","Type":"ContainerStarted","Data":"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"}
Oct 30 22:22:37.689274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.689237    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" event={"ID":"69a06f34-f671-49f3-961c-45adef24efac","Type":"ContainerStarted","Data":"6eae4235a3604a20ab6965a28546587458308492a7c8eba726c1f1e3862359ff"}
Oct 30 22:22:37.814268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.814208    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podStartSLOduration=1.8141696170000001 podStartE2EDuration="1.814169617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:22:37.813108095 +0000 UTC m=+2680.986432522" watchObservedRunningTime="2026-10-30 22:22:37.814169617 +0000 UTC m=+2680.987494043"
Oct 30 22:22:37.914385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:37.914350    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:38.916413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:38.916374    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:38.916850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:38.916438    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:39.919270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:39.914414    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:39.919270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:39.914480    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:40.915636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:40.914612    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:40.915636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:40.914675    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:41.914385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:41.914347    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:41.914881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:41.914415    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:42.652707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:42.652672    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:22:42.914599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:42.914488    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:42.915208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:42.915168    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:43.914251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:43.914213    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused" start-of-body=
Oct 30 22:22:43.914452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:43.914275    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.215:15021/healthz/ready\": dial tcp 10.132.0.215:15021: connect: connection refused"
Oct 30 22:22:44.666572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:44.666535    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:22:44.906498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:44.906456    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:22:44.921361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:44.921267    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:44.921859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:44.921828    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:44.922499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:44.922476    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:22:50.811064 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:22:50.811025    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:22:57.311245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:57.311215    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:22:57.325456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:22:57.325427    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:23:20.812521 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:23:20.812479    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:23:22.268497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:22.268460    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:23:30.507172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.507138    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"]
Oct 30 22:23:30.511562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.511541    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.538006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.537976    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.538166 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.538040    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.538261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.538237    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n47v2\" (UniqueName: \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.565487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.565454    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"]
Oct 30 22:23:30.612367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.612336    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:23:30.612538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.612403    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:23:30.639038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.639004    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n47v2\" (UniqueName: \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.639211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.639116    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.639211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.639161    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.639834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.639811    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.641512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.641489    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.701049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.701013    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n47v2\" (UniqueName: \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\") pod \"maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:30.826271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:30.826182    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:31.201244 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:23:31.201199    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode9cd5c26_0cea_4277_8884_9327ae27bdc2.slice/crio-cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b WatchSource:0}: Error finding container cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b: Status 404 returned error can't find the container with id cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b
Oct 30 22:23:31.204209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:31.204176    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"]
Oct 30 22:23:32.066799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:32.066763    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t" event={"ID":"e9cd5c26-0cea-4277-8884-9327ae27bdc2","Type":"ContainerStarted","Data":"38ddc3447410386a7cf3c13e67c591a2887d2c3c6bbbdfb9db038a93d58472d4"}
Oct 30 22:23:32.067203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:32.066821    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t" event={"ID":"e9cd5c26-0cea-4277-8884-9327ae27bdc2","Type":"ContainerStarted","Data":"cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b"}
Oct 30 22:23:32.185136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:32.185082    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t" podStartSLOduration=2.185067684 podStartE2EDuration="2.185067684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:23:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:23:32.183920044 +0000 UTC m=+2735.357244464" watchObservedRunningTime="2026-10-30 22:23:32.185067684 +0000 UTC m=+2735.358392111"
Oct 30 22:23:33.075485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:33.075427    4882 generic.go:356] "Generic (PLEG): container finished" podID="e9cd5c26-0cea-4277-8884-9327ae27bdc2" containerID="38ddc3447410386a7cf3c13e67c591a2887d2c3c6bbbdfb9db038a93d58472d4" exitCode=0
Oct 30 22:23:33.076118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:33.075512    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t" event={"ID":"e9cd5c26-0cea-4277-8884-9327ae27bdc2","Type":"ContainerDied","Data":"38ddc3447410386a7cf3c13e67c591a2887d2c3c6bbbdfb9db038a93d58472d4"}
Oct 30 22:23:34.235746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.235719    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:34.282032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.281997    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\") pod \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") "
Oct 30 22:23:34.282222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.282090    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\") pod \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") "
Oct 30 22:23:34.282295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.282253    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\" (UniqueName: \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\") pod \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\" (UID: \"e9cd5c26-0cea-4277-8884-9327ae27bdc2\") "
Oct 30 22:23:34.282544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.282507    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca" pod "e9cd5c26-0cea-4277-8884-9327ae27bdc2" (UID: "e9cd5c26-0cea-4277-8884-9327ae27bdc2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:34.282885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.282868    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.284322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.284288    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca" pod "e9cd5c26-0cea-4277-8884-9327ae27bdc2" (UID: "e9cd5c26-0cea-4277-8884-9327ae27bdc2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:23:34.284416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.284348    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2" pod "e9cd5c26-0cea-4277-8884-9327ae27bdc2" (UID: "e9cd5c26-0cea-4277-8884-9327ae27bdc2"). InnerVolumeSpecName "kube-api-access-n47v2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:34.384632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.384510    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n47v2\" (UniqueName: \"kubernetes.io/projected/e9cd5c26-0cea-4277-8884-9327ae27bdc2-kube-api-access-n47v2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:34.384632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:34.384552    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e9cd5c26-0cea-4277-8884-9327ae27bdc2-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:35.091493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:35.091458    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t" event={"ID":"e9cd5c26-0cea-4277-8884-9327ae27bdc2","Type":"ContainerDied","Data":"cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b"}
Oct 30 22:23:35.091682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:35.091510    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6f5e20c04937cff6d49820cb7ebedb6e8a1ae6c1e2c0b8c030eafc4d52e95b"
Oct 30 22:23:35.091682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:35.091508    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"
Oct 30 22:23:42.044038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:42.044002    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:23:42.157996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:42.157963    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:23:47.044535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.044496    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" podUID="825553dd-6d58-4bc7-a297-c5ae0a74647e" containerName="payload-processing" containerID="cri-o://075be162225d24979bd5d7f4a88c83b7c9d116da66455f94a7eebc33939b32ef" gracePeriod=25
Oct 30 22:23:47.159289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.159246    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" podUID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" containerName="payload-pre-processing" containerID="cri-o://ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd" gracePeriod=25
Oct 30 22:23:47.172464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.172430    4882 generic.go:356] "Generic (PLEG): container finished" podID="825553dd-6d58-4bc7-a297-c5ae0a74647e" containerID="075be162225d24979bd5d7f4a88c83b7c9d116da66455f94a7eebc33939b32ef" exitCode=0
Oct 30 22:23:47.172642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.172487    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" event={"ID":"825553dd-6d58-4bc7-a297-c5ae0a74647e","Type":"ContainerDied","Data":"075be162225d24979bd5d7f4a88c83b7c9d116da66455f94a7eebc33939b32ef"}
Oct 30 22:23:47.332243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.332214    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:23:47.445544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.445511    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:23:47.450051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.450024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\" (UniqueName: \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\") pod \"825553dd-6d58-4bc7-a297-c5ae0a74647e\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") "
Oct 30 22:23:47.450485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.450218    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\") pod \"825553dd-6d58-4bc7-a297-c5ae0a74647e\" (UID: \"825553dd-6d58-4bc7-a297-c5ae0a74647e\") "
Oct 30 22:23:47.451107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.451080    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume" pod "825553dd-6d58-4bc7-a297-c5ae0a74647e" (UID: "825553dd-6d58-4bc7-a297-c5ae0a74647e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:47.452661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.452621    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt" pod "825553dd-6d58-4bc7-a297-c5ae0a74647e" (UID: "825553dd-6d58-4bc7-a297-c5ae0a74647e"). InnerVolumeSpecName "kube-api-access-4p7jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:47.551493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.551456    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\") pod \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") "
Oct 30 22:23:47.551725 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.551696    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\" (UniqueName: \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\") pod \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\" (UID: \"8dcac791-17ec-481d-8ce5-59ebbc9f94f1\") "
Oct 30 22:23:47.551957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.551931    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume" pod "8dcac791-17ec-481d-8ce5-59ebbc9f94f1" (UID: "8dcac791-17ec-481d-8ce5-59ebbc9f94f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:23:47.552163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.552145    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4p7jt\" (UniqueName: \"kubernetes.io/projected/825553dd-6d58-4bc7-a297-c5ae0a74647e-kube-api-access-4p7jt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:47.552225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.552168    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:47.552225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.552184    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/825553dd-6d58-4bc7-a297-c5ae0a74647e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:47.553859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.553830    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8" pod "8dcac791-17ec-481d-8ce5-59ebbc9f94f1" (UID: "8dcac791-17ec-481d-8ce5-59ebbc9f94f1"). InnerVolumeSpecName "kube-api-access-7vdr8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:23:47.653269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:47.653231    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vdr8\" (UniqueName: \"kubernetes.io/projected/8dcac791-17ec-481d-8ce5-59ebbc9f94f1-kube-api-access-7vdr8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:23:48.179809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.179779    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"
Oct 30 22:23:48.179809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.179795    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr" event={"ID":"825553dd-6d58-4bc7-a297-c5ae0a74647e","Type":"ContainerDied","Data":"88e3f59a1004a2b3a9517dd64a80084696c07a26563c57e34b39d761d078f40b"}
Oct 30 22:23:48.180340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.179885    4882 scope.go:122] "RemoveContainer" containerID="075be162225d24979bd5d7f4a88c83b7c9d116da66455f94a7eebc33939b32ef"
Oct 30 22:23:48.181489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.181461    4882 generic.go:356] "Generic (PLEG): container finished" podID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" containerID="ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd" exitCode=0
Oct 30 22:23:48.181579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.181515    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"
Oct 30 22:23:48.181579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.181552    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" event={"ID":"8dcac791-17ec-481d-8ce5-59ebbc9f94f1","Type":"ContainerDied","Data":"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"}
Oct 30 22:23:48.181713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.181621    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft" event={"ID":"8dcac791-17ec-481d-8ce5-59ebbc9f94f1","Type":"ContainerDied","Data":"f5e2fcad48ab9bc0eb6f6af3c995b1fc4d332adb8de137faab322ff4e3737ea7"}
Oct 30 22:23:48.193507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.193475    4882 scope.go:122] "RemoveContainer" containerID="ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"
Oct 30 22:23:48.204767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.204742    4882 scope.go:122] "RemoveContainer" containerID="ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"
Oct 30 22:23:48.205050 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:23:48.205028    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd\": container with ID starting with ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd not found: ID does not exist" containerID="ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"
Oct 30 22:23:48.205113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.205061    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd"} err="failed to get container status \"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd\": rpc error: code = NotFound desc = could not find container \"ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd\": container with ID starting with ab4f0e7121272194d16ffd0f0630c0bb875769b48d8aa635171bb48f5f353dfd not found: ID does not exist"
Oct 30 22:23:48.348021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.347979    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:23:48.354795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.354765    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-3cecc7-58766fc779-v6psr"]
Oct 30 22:23:48.467136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.467103    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:23:48.470429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.470397    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-3cecc7-fd5cb8f47-f84ft"]
Oct 30 22:23:48.943091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.943056    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="825553dd-6d58-4bc7-a297-c5ae0a74647e" path="/var/lib/kubelet/pods/825553dd-6d58-4bc7-a297-c5ae0a74647e/volumes"
Oct 30 22:23:48.943495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:48.943482    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" path="/var/lib/kubelet/pods/8dcac791-17ec-481d-8ce5-59ebbc9f94f1/volumes"
Oct 30 22:23:50.809865 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:23:50.809821    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:23:58.430739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:58.430694    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:23:58.847500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:58.847470    4882 scope.go:122] "RemoveContainer" containerID="9b47f7af9b6588a39464c03ddd3f5092a98f920a3c5482f12344afa3aa4aa613"
Oct 30 22:23:58.857529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:23:58.857503    4882 scope.go:122] "RemoveContainer" containerID="9a8ce8a87a868bfb0c505085725faa6e27803017c13c90fd1eb347df320844ee"
Oct 30 22:24:06.879868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.879835    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"]
Oct 30 22:24:06.880438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880425    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" containerName="payload-pre-processing"
Oct 30 22:24:06.880497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880440    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" containerName="payload-pre-processing"
Oct 30 22:24:06.880497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880456    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e9cd5c26-0cea-4277-8884-9327ae27bdc2" containerName="revoke-keys"
Oct 30 22:24:06.880497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880461    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e9cd5c26-0cea-4277-8884-9327ae27bdc2" containerName="revoke-keys"
Oct 30 22:24:06.880497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880470    4882 cpu_manager.go:414] "containerMap: removing container" podUID="825553dd-6d58-4bc7-a297-c5ae0a74647e" containerName="payload-processing"
Oct 30 22:24:06.880497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880475    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="825553dd-6d58-4bc7-a297-c5ae0a74647e" containerName="payload-processing"
Oct 30 22:24:06.880672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880573    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8dcac791-17ec-481d-8ce5-59ebbc9f94f1" containerName="payload-pre-processing"
Oct 30 22:24:06.880672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880581    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e9cd5c26-0cea-4277-8884-9327ae27bdc2" containerName="revoke-keys"
Oct 30 22:24:06.880672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.880599    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="825553dd-6d58-4bc7-a297-c5ae0a74647e" containerName="payload-processing"
Oct 30 22:24:06.884512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.884495    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:06.946320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.946290    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"]
Oct 30 22:24:06.966229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.966194    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:06.966408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.966352    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:06.966452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.966424    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h9f95\" (UniqueName: \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:06.985325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.985297    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:06.985325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:06.985316    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:24:07.067960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.067927    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h9f95\" (UniqueName: \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.068186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.068033    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.068186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.068106    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.068753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.068732    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.070463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.070445    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.130393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.130325    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h9f95\" (UniqueName: \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\") pod \"maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.198664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.198629    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:07.580393 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:24:07.580351    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f3cad3b_23c2_4090_a6cb_e4232e0fdc09.slice/crio-e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6 WatchSource:0}: Error finding container e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6: Status 404 returned error can't find the container with id e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6
Oct 30 22:24:07.582526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:07.582497    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"]
Oct 30 22:24:08.316831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:08.316728    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb" event={"ID":"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09","Type":"ContainerStarted","Data":"4924fd7181c60a0f7d51a8613a874bad032485a2cc33533329a32f48475fed58"}
Oct 30 22:24:08.316831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:08.316815    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb" event={"ID":"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09","Type":"ContainerStarted","Data":"e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6"}
Oct 30 22:24:08.435229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:08.435164    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb" podStartSLOduration=2.435144805 podStartE2EDuration="2.435144805s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:08.433648111 +0000 UTC m=+2771.606972552" watchObservedRunningTime="2026-10-30 22:24:08.435144805 +0000 UTC m=+2771.608469233"
Oct 30 22:24:09.324728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:09.324688    4882 generic.go:356] "Generic (PLEG): container finished" podID="0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" containerID="4924fd7181c60a0f7d51a8613a874bad032485a2cc33533329a32f48475fed58" exitCode=0
Oct 30 22:24:09.325215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:09.324773    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb" event={"ID":"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09","Type":"ContainerDied","Data":"4924fd7181c60a0f7d51a8613a874bad032485a2cc33533329a32f48475fed58"}
Oct 30 22:24:10.478254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.478227    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:10.609801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.609715    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\") pod \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") "
Oct 30 22:24:10.610169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.609912    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\") pod \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") "
Oct 30 22:24:10.610169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.610040    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\" (UniqueName: \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\") pod \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\" (UID: \"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09\") "
Oct 30 22:24:10.612291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.612255    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95" pod "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" (UID: "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09"). InnerVolumeSpecName "kube-api-access-h9f95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:10.612404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.612348    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca" pod "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" (UID: "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:10.612958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.612930    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca" pod "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" (UID: "0f3cad3b-23c2-4090-a6cb-e4232e0fdc09"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:10.711650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.711581    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.711650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.711645    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h9f95\" (UniqueName: \"kubernetes.io/projected/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-kube-api-access-h9f95\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.711650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.711659    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:10.867301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.867217    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"]
Oct 30 22:24:10.876020 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.875989    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-3cecc7-qm46t"]
Oct 30 22:24:10.943896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:10.943862    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e9cd5c26-0cea-4277-8884-9327ae27bdc2" path="/var/lib/kubelet/pods/e9cd5c26-0cea-4277-8884-9327ae27bdc2/volumes"
Oct 30 22:24:11.338346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:11.338318    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"
Oct 30 22:24:11.338497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:11.338352    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb" event={"ID":"0f3cad3b-23c2-4090-a6cb-e4232e0fdc09","Type":"ContainerDied","Data":"e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6"}
Oct 30 22:24:11.338497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:11.338399    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e546d746e5bf28323562c13a71df9c40ebb854f6f0fe0eeb9768915e22a121c6"
Oct 30 22:24:18.487555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.487518    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"]
Oct 30 22:24:18.488133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.488118    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" containerName="revoke-keys"
Oct 30 22:24:18.488180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.488135    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" containerName="revoke-keys"
Oct 30 22:24:18.488270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.488260    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" containerName="revoke-keys"
Oct 30 22:24:18.491891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.491873    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.546046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.546011    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"]
Oct 30 22:24:18.592956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.592919    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:18.593173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.593038    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:24:18.599625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.599565    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bh7ht\" (UniqueName: \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.599755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.599683    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.599864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.599841    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.701199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.701161    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.701400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.701268    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.701400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.701329    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bh7ht\" (UniqueName: \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.701889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.701865    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.703750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.703719    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.763449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.763378    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bh7ht\" (UniqueName: \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\") pod \"maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:18.810843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:18.810796    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:19.184533 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:24:19.184497    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod507b517a_7d9d_497a_a54f_ea41e5509e64.slice/crio-b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731 WatchSource:0}: Error finding container b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731: Status 404 returned error can't find the container with id b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731
Oct 30 22:24:19.186990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:19.186959    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"]
Oct 30 22:24:19.389328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:19.389286    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6" event={"ID":"507b517a-7d9d-497a-a54f-ea41e5509e64","Type":"ContainerStarted","Data":"b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731"}
Oct 30 22:24:20.396884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:20.396853    4882 generic.go:356] "Generic (PLEG): container finished" podID="507b517a-7d9d-497a-a54f-ea41e5509e64" containerID="3cfcef73a45e53a11b5d5c148d7c73039b23eecf8a9f2303d4bb162a1dffab23" exitCode=0
Oct 30 22:24:20.397302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:20.396936    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6" event={"ID":"507b517a-7d9d-497a-a54f-ea41e5509e64","Type":"ContainerDied","Data":"3cfcef73a45e53a11b5d5c148d7c73039b23eecf8a9f2303d4bb162a1dffab23"}
Oct 30 22:24:20.807801 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:20.807680    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:24:21.554550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.554522    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:21.639033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.638998    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\") pod \"507b517a-7d9d-497a-a54f-ea41e5509e64\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") "
Oct 30 22:24:21.639227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.639057    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\") pod \"507b517a-7d9d-497a-a54f-ea41e5509e64\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") "
Oct 30 22:24:21.639295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.639251    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\" (UniqueName: \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\") pod \"507b517a-7d9d-497a-a54f-ea41e5509e64\" (UID: \"507b517a-7d9d-497a-a54f-ea41e5509e64\") "
Oct 30 22:24:21.639471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.639447    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca" pod "507b517a-7d9d-497a-a54f-ea41e5509e64" (UID: "507b517a-7d9d-497a-a54f-ea41e5509e64"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:21.639687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.639669    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:21.641159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.641127    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca" pod "507b517a-7d9d-497a-a54f-ea41e5509e64" (UID: "507b517a-7d9d-497a-a54f-ea41e5509e64"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:21.641318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.641294    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht" pod "507b517a-7d9d-497a-a54f-ea41e5509e64" (UID: "507b517a-7d9d-497a-a54f-ea41e5509e64"). InnerVolumeSpecName "kube-api-access-bh7ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:21.740882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.740799    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/507b517a-7d9d-497a-a54f-ea41e5509e64-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:21.740882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:21.740829    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bh7ht\" (UniqueName: \"kubernetes.io/projected/507b517a-7d9d-497a-a54f-ea41e5509e64-kube-api-access-bh7ht\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:22.228995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:22.228963    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:24:22.339116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:22.339056    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:24:22.413822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:22.413788    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"
Oct 30 22:24:22.413994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:22.413820    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6" event={"ID":"507b517a-7d9d-497a-a54f-ea41e5509e64","Type":"ContainerDied","Data":"b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731"}
Oct 30 22:24:22.413994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:22.413888    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b49b569f458bfff84510823774b50e38c70334aab3e54d7fceac9bebcf9ce731"
Oct 30 22:24:27.229814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.229771    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" podUID="01760d91-50ba-4479-ab47-9f0750d6b3b0" containerName="payload-processing" containerID="cri-o://981d89a917f060869123c3ce663103b75470947a5ce185f3d0e335e5cf2b2173" gracePeriod=25
Oct 30 22:24:27.339802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.339751    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" podUID="ecc947cb-5986-412d-9399-d075c49b868f" containerName="payload-pre-processing" containerID="cri-o://8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813" gracePeriod=25
Oct 30 22:24:27.449281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.449246    4882 generic.go:356] "Generic (PLEG): container finished" podID="01760d91-50ba-4479-ab47-9f0750d6b3b0" containerID="981d89a917f060869123c3ce663103b75470947a5ce185f3d0e335e5cf2b2173" exitCode=0
Oct 30 22:24:27.449532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.449506    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" event={"ID":"01760d91-50ba-4479-ab47-9f0750d6b3b0","Type":"ContainerDied","Data":"981d89a917f060869123c3ce663103b75470947a5ce185f3d0e335e5cf2b2173"}
Oct 30 22:24:27.509057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.509027    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:24:27.607015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.606983    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\" (UniqueName: \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\") pod \"01760d91-50ba-4479-ab47-9f0750d6b3b0\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") "
Oct 30 22:24:27.607206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.607192    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\") pod \"01760d91-50ba-4479-ab47-9f0750d6b3b0\" (UID: \"01760d91-50ba-4479-ab47-9f0750d6b3b0\") "
Oct 30 22:24:27.607653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.607621    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume" pod "01760d91-50ba-4479-ab47-9f0750d6b3b0" (UID: "01760d91-50ba-4479-ab47-9f0750d6b3b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:27.609228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.609201    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx" pod "01760d91-50ba-4479-ab47-9f0750d6b3b0" (UID: "01760d91-50ba-4479-ab47-9f0750d6b3b0"). InnerVolumeSpecName "kube-api-access-4ckxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:27.611082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.611062    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:24:27.708776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.708745    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\" (UniqueName: \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\") pod \"ecc947cb-5986-412d-9399-d075c49b868f\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") "
Oct 30 22:24:27.708962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.708950    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\") pod \"ecc947cb-5986-412d-9399-d075c49b868f\" (UID: \"ecc947cb-5986-412d-9399-d075c49b868f\") "
Oct 30 22:24:27.709339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.709319    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01760d91-50ba-4479-ab47-9f0750d6b3b0-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:27.709400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.709339    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume" pod "ecc947cb-5986-412d-9399-d075c49b868f" (UID: "ecc947cb-5986-412d-9399-d075c49b868f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:27.709400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.709347    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ckxx\" (UniqueName: \"kubernetes.io/projected/01760d91-50ba-4479-ab47-9f0750d6b3b0-kube-api-access-4ckxx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:27.710900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.710878    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r" pod "ecc947cb-5986-412d-9399-d075c49b868f" (UID: "ecc947cb-5986-412d-9399-d075c49b868f"). InnerVolumeSpecName "kube-api-access-mhp6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:27.811079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.810991    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ecc947cb-5986-412d-9399-d075c49b868f-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:27.811079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:27.811024    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mhp6r\" (UniqueName: \"kubernetes.io/projected/ecc947cb-5986-412d-9399-d075c49b868f-kube-api-access-mhp6r\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:28.457072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.457036    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"
Oct 30 22:24:28.457558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.457042    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj" event={"ID":"01760d91-50ba-4479-ab47-9f0750d6b3b0","Type":"ContainerDied","Data":"b7e4e62fde24b7c6a91f7dcfac0bcfbbceab2020a984223b6a9b1b30cd0779b1"}
Oct 30 22:24:28.457558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.457226    4882 scope.go:122] "RemoveContainer" containerID="981d89a917f060869123c3ce663103b75470947a5ce185f3d0e335e5cf2b2173"
Oct 30 22:24:28.458902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.458873    4882 generic.go:356] "Generic (PLEG): container finished" podID="ecc947cb-5986-412d-9399-d075c49b868f" containerID="8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813" exitCode=0
Oct 30 22:24:28.459034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.458929    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"
Oct 30 22:24:28.459034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.458950    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" event={"ID":"ecc947cb-5986-412d-9399-d075c49b868f","Type":"ContainerDied","Data":"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"}
Oct 30 22:24:28.459034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.459016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k" event={"ID":"ecc947cb-5986-412d-9399-d075c49b868f","Type":"ContainerDied","Data":"9fbfc726069b04b12db1429d3e10e773733db83ddb8aa96e9380c91cb6b8fa3d"}
Oct 30 22:24:28.468357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.468335    4882 scope.go:122] "RemoveContainer" containerID="8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"
Oct 30 22:24:28.478429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.478405    4882 scope.go:122] "RemoveContainer" containerID="8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"
Oct 30 22:24:28.478740 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:28.478720    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813\": container with ID starting with 8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813 not found: ID does not exist" containerID="8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"
Oct 30 22:24:28.478827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.478749    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813"} err="failed to get container status \"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813\": rpc error: code = NotFound desc = could not find container \"8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813\": container with ID starting with 8af2e7889fc3b403063968055a392b1e1dcefcf99a2bdc068d95553ce0814813 not found: ID does not exist"
Oct 30 22:24:28.625345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.625310    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:24:28.675265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.675229    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-ca6d0b-6968c74b4c-qvhdj"]
Oct 30 22:24:28.785383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.785343    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:24:28.788254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.788219    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-ca6d0b-5945897d99-27l2k"]
Oct 30 22:24:28.943772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.943701    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01760d91-50ba-4479-ab47-9f0750d6b3b0" path="/var/lib/kubelet/pods/01760d91-50ba-4479-ab47-9f0750d6b3b0/volumes"
Oct 30 22:24:28.944123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:28.944110    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecc947cb-5986-412d-9399-d075c49b868f" path="/var/lib/kubelet/pods/ecc947cb-5986-412d-9399-d075c49b868f/volumes"
Oct 30 22:24:29.165648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:29.165616    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml_36be8729-107f-4459-b3f2-f4f06d6a90a9/payload-processing/0.log"
Oct 30 22:24:29.576670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:29.576633    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:24:29.635289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:29.635254    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:24:30.125656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:30.125631    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59_23c02225-8584-47fe-8e58-f7ff75b8aa26/payload-processing/0.log"
Oct 30 22:24:30.371144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:30.371118    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59_23c02225-8584-47fe-8e58-f7ff75b8aa26/payload-processing/0.log"
Oct 30 22:24:31.435047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435009    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"]
Oct 30 22:24:31.435628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435612    4882 cpu_manager.go:414] "containerMap: removing container" podUID="01760d91-50ba-4479-ab47-9f0750d6b3b0" containerName="payload-processing"
Oct 30 22:24:31.435679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435629    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01760d91-50ba-4479-ab47-9f0750d6b3b0" containerName="payload-processing"
Oct 30 22:24:31.435679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435659    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ecc947cb-5986-412d-9399-d075c49b868f" containerName="payload-pre-processing"
Oct 30 22:24:31.435679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435666    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecc947cb-5986-412d-9399-d075c49b868f" containerName="payload-pre-processing"
Oct 30 22:24:31.435783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435702    4882 cpu_manager.go:414] "containerMap: removing container" podUID="507b517a-7d9d-497a-a54f-ea41e5509e64" containerName="revoke-keys"
Oct 30 22:24:31.435783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435708    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="507b517a-7d9d-497a-a54f-ea41e5509e64" containerName="revoke-keys"
Oct 30 22:24:31.435892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435814    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="01760d91-50ba-4479-ab47-9f0750d6b3b0" containerName="payload-processing"
Oct 30 22:24:31.435892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435826    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecc947cb-5986-412d-9399-d075c49b868f" containerName="payload-pre-processing"
Oct 30 22:24:31.435892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.435838    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="507b517a-7d9d-497a-a54f-ea41e5509e64" containerName="revoke-keys"
Oct 30 22:24:31.440821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.440802    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.492745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.492715    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"]
Oct 30 22:24:31.539469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.539442    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:31.539696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.539551    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:24:31.559863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.559834    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.560038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.559880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp8ht\" (UniqueName: \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.560105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.560044    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.661723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.661686    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.661949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.661807    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.661949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.661862    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp8ht\" (UniqueName: \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.662487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.662460    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.664255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.664234    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.723966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.723866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp8ht\" (UniqueName: \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\") pod \"maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:31.754698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:31.754664    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:32.130818 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:24:32.130787    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod660b7eb8_0cb8_4c37_8fbc_85ea7bdf8f28.slice/crio-9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd WatchSource:0}: Error finding container 9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd: Status 404 returned error can't find the container with id 9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd
Oct 30 22:24:32.133581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:32.133520    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"]
Oct 30 22:24:32.487303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:32.487265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw" event={"ID":"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28","Type":"ContainerStarted","Data":"1a282cfcd0980918a51a113875f10570d2260c0f8418044896b2b3fc05062a1f"}
Oct 30 22:24:32.487715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:32.487327    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw" event={"ID":"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28","Type":"ContainerStarted","Data":"9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd"}
Oct 30 22:24:33.495933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:33.495894    4882 generic.go:356] "Generic (PLEG): container finished" podID="660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" containerID="1a282cfcd0980918a51a113875f10570d2260c0f8418044896b2b3fc05062a1f" exitCode=0
Oct 30 22:24:33.496344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:33.495981    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw" event={"ID":"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28","Type":"ContainerDied","Data":"1a282cfcd0980918a51a113875f10570d2260c0f8418044896b2b3fc05062a1f"}
Oct 30 22:24:34.577618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.577526    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" podUID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" containerName="payload-processing" containerID="cri-o://1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c" gracePeriod=25
Oct 30 22:24:34.601082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.601050    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"]
Oct 30 22:24:34.636553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.636488    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" podUID="d9c7c856-0801-4bf1-a454-07adbdd6736e" containerName="payload-pre-processing" containerID="cri-o://c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b" gracePeriod=25
Oct 30 22:24:34.650343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.650311    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-85d00c-wrzn6"]
Oct 30 22:24:34.750064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.749925    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:34.803567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.803486    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\") pod \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") "
Oct 30 22:24:34.803567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.803532    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\" (UniqueName: \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\") pod \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") "
Oct 30 22:24:34.803817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.803722    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\") pod \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\" (UID: \"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28\") "
Oct 30 22:24:34.804229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.804196    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca" pod "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" (UID: "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:34.804384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.804365    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:34.808155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.808127    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht" pod "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" (UID: "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28"). InnerVolumeSpecName "kube-api-access-lp8ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:34.808939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.808907    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca" pod "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" (UID: "660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:34.905880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.905835    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:34.905880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.905878    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp8ht\" (UniqueName: \"kubernetes.io/projected/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28-kube-api-access-lp8ht\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:34.937963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.937937    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:24:34.941533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.941509    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:24:34.943210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:34.943186    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="507b517a-7d9d-497a-a54f-ea41e5509e64" path="/var/lib/kubelet/pods/507b517a-7d9d-497a-a54f-ea41e5509e64/volumes"
Oct 30 22:24:35.007121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007089    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\" (UniqueName: \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\") pod \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") "
Oct 30 22:24:35.007305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007168    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\") pod \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\" (UID: \"eb57a61d-6086-4d2c-8d18-66eaec59d0bd\") "
Oct 30 22:24:35.007305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007245    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\") pod \"d9c7c856-0801-4bf1-a454-07adbdd6736e\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") "
Oct 30 22:24:35.007305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007304    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\" (UniqueName: \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\") pod \"d9c7c856-0801-4bf1-a454-07adbdd6736e\" (UID: \"d9c7c856-0801-4bf1-a454-07adbdd6736e\") "
Oct 30 22:24:35.007611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007562    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume" pod "eb57a61d-6086-4d2c-8d18-66eaec59d0bd" (UID: "eb57a61d-6086-4d2c-8d18-66eaec59d0bd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.007730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007656    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume" pod "d9c7c856-0801-4bf1-a454-07adbdd6736e" (UID: "d9c7c856-0801-4bf1-a454-07adbdd6736e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:35.007794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.007760    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.009375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.009352    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs" pod "d9c7c856-0801-4bf1-a454-07adbdd6736e" (UID: "d9c7c856-0801-4bf1-a454-07adbdd6736e"). InnerVolumeSpecName "kube-api-access-wprxs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.009472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.009373    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb" pod "eb57a61d-6086-4d2c-8d18-66eaec59d0bd" (UID: "eb57a61d-6086-4d2c-8d18-66eaec59d0bd"). InnerVolumeSpecName "kube-api-access-zjxrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:35.109072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.109040    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d9c7c856-0801-4bf1-a454-07adbdd6736e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.109072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.109069    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wprxs\" (UniqueName: \"kubernetes.io/projected/d9c7c856-0801-4bf1-a454-07adbdd6736e-kube-api-access-wprxs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.109307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.109130    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjxrb\" (UniqueName: \"kubernetes.io/projected/eb57a61d-6086-4d2c-8d18-66eaec59d0bd-kube-api-access-zjxrb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:35.510863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.510773    4882 generic.go:356] "Generic (PLEG): container finished" podID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" containerID="1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c" exitCode=0
Oct 30 22:24:35.510863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.510821    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"
Oct 30 22:24:35.510863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.510848    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" event={"ID":"eb57a61d-6086-4d2c-8d18-66eaec59d0bd","Type":"ContainerDied","Data":"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"}
Oct 30 22:24:35.511147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.510907    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd" event={"ID":"eb57a61d-6086-4d2c-8d18-66eaec59d0bd","Type":"ContainerDied","Data":"4231725c1ab96aeff21d1eec3319d7db582c75121c5eb4f4f8d1f2859af15f4e"}
Oct 30 22:24:35.511147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.510933    4882 scope.go:122] "RemoveContainer" containerID="1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"
Oct 30 22:24:35.512566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.512540    4882 generic.go:356] "Generic (PLEG): container finished" podID="d9c7c856-0801-4bf1-a454-07adbdd6736e" containerID="c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b" exitCode=0
Oct 30 22:24:35.512700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.512685    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" event={"ID":"d9c7c856-0801-4bf1-a454-07adbdd6736e","Type":"ContainerDied","Data":"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"}
Oct 30 22:24:35.512747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.512724    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc" event={"ID":"d9c7c856-0801-4bf1-a454-07adbdd6736e","Type":"ContainerDied","Data":"cd8f65e34c45f2e1b3dfee1966e09b67a1103d0b93d674a873dec5fc3a1ad7fc"}
Oct 30 22:24:35.512790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.512775    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"
Oct 30 22:24:35.514577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.514552    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw" event={"ID":"660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28","Type":"ContainerDied","Data":"9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd"}
Oct 30 22:24:35.514686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.514615    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9635bf8b94aef994946f57a48560c23a9d6b4249b31784f3a610cc9d16dadbdd"
Oct 30 22:24:35.514686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.514671    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"
Oct 30 22:24:35.523115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.523095    4882 scope.go:122] "RemoveContainer" containerID="1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"
Oct 30 22:24:35.523400 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:35.523381    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c\": container with ID starting with 1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c not found: ID does not exist" containerID="1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"
Oct 30 22:24:35.523446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.523408    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c"} err="failed to get container status \"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c\": rpc error: code = NotFound desc = could not find container \"1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c\": container with ID starting with 1a5c159362028b7527df8e5eafa59e99659cd3fa24f96e85b0cbacc8d1c1f44c not found: ID does not exist"
Oct 30 22:24:35.523446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.523425    4882 scope.go:122] "RemoveContainer" containerID="c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"
Oct 30 22:24:35.534128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.534106    4882 scope.go:122] "RemoveContainer" containerID="c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"
Oct 30 22:24:35.534406 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:35.534390    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b\": container with ID starting with c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b not found: ID does not exist" containerID="c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"
Oct 30 22:24:35.534456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.534415    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b"} err="failed to get container status \"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b\": rpc error: code = NotFound desc = could not find container \"c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b\": container with ID starting with c2cc2002d91e68222186a034a4b7549c5e5ae9daac13388cdb7fe28d2ec9093b not found: ID does not exist"
Oct 30 22:24:35.702510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.702479    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:24:35.705603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.705557    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-85d00c-7b469766d7-zcrgc"]
Oct 30 22:24:35.873906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.873872    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:24:35.878915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:35.878884    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-85d00c-58cf887765-d2kzd"]
Oct 30 22:24:36.944408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:36.944333    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9c7c856-0801-4bf1-a454-07adbdd6736e" path="/var/lib/kubelet/pods/d9c7c856-0801-4bf1-a454-07adbdd6736e/volumes"
Oct 30 22:24:36.944779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:36.944759    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" path="/var/lib/kubelet/pods/eb57a61d-6086-4d2c-8d18-66eaec59d0bd/volumes"
Oct 30 22:24:39.265648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.265611    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"]
Oct 30 22:24:39.266175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266160    4882 cpu_manager.go:414] "containerMap: removing container" podUID="660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" containerName="revoke-keys"
Oct 30 22:24:39.266233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266176    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" containerName="revoke-keys"
Oct 30 22:24:39.266233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266208    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d9c7c856-0801-4bf1-a454-07adbdd6736e" containerName="payload-pre-processing"
Oct 30 22:24:39.266233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266213    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9c7c856-0801-4bf1-a454-07adbdd6736e" containerName="payload-pre-processing"
Oct 30 22:24:39.266345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266237    4882 cpu_manager.go:414] "containerMap: removing container" podUID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" containerName="payload-processing"
Oct 30 22:24:39.266345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266242    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" containerName="payload-processing"
Oct 30 22:24:39.266410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266346    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9c7c856-0801-4bf1-a454-07adbdd6736e" containerName="payload-pre-processing"
Oct 30 22:24:39.266410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266358    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb57a61d-6086-4d2c-8d18-66eaec59d0bd" containerName="payload-processing"
Oct 30 22:24:39.266410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.266367    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" containerName="revoke-keys"
Oct 30 22:24:39.271382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.271360    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.325602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.325556    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"]
Oct 30 22:24:39.358708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.358677    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.358872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.358746    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4bmdl\" (UniqueName: \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.358872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.358859    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.371121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.371092    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:39.371227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.371123    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:24:39.460200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.460160    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4bmdl\" (UniqueName: \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.460403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.460247    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.460403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.460342    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.460990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.460970    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.462601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.462568    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.525350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.525271    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4bmdl\" (UniqueName: \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\") pod \"maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.585189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.585157    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:39.959339 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:24:39.959310    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4645fb6e_f9d9_487f_9aa8_920249e2e270.slice/crio-07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff WatchSource:0}: Error finding container 07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff: Status 404 returned error can't find the container with id 07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff
Oct 30 22:24:39.961941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:39.961914    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"]
Oct 30 22:24:40.549880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:40.549781    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz" event={"ID":"4645fb6e-f9d9-487f-9aa8-920249e2e270","Type":"ContainerStarted","Data":"501dd1f1cbc9857b4a9af96b1a7e8c3eb0d466e38e3857660d3dd635f65c1795"}
Oct 30 22:24:40.549880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:40.549836    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz" event={"ID":"4645fb6e-f9d9-487f-9aa8-920249e2e270","Type":"ContainerStarted","Data":"07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff"}
Oct 30 22:24:40.668954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:40.668887    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz" podStartSLOduration=1.668869338 podStartE2EDuration="1.668869338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:40.666573825 +0000 UTC m=+2803.839898252" watchObservedRunningTime="2026-10-30 22:24:40.668869338 +0000 UTC m=+2803.842193765"
Oct 30 22:24:41.558772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:41.558735    4882 generic.go:356] "Generic (PLEG): container finished" podID="4645fb6e-f9d9-487f-9aa8-920249e2e270" containerID="501dd1f1cbc9857b4a9af96b1a7e8c3eb0d466e38e3857660d3dd635f65c1795" exitCode=0
Oct 30 22:24:41.559228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:41.558781    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz" event={"ID":"4645fb6e-f9d9-487f-9aa8-920249e2e270","Type":"ContainerDied","Data":"501dd1f1cbc9857b4a9af96b1a7e8c3eb0d466e38e3857660d3dd635f65c1795"}
Oct 30 22:24:41.667620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:41.667573    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:24:41.667806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:41.667786    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" containerID="cri-o://6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898" gracePeriod=30
Oct 30 22:24:42.718052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.718023    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:42.799504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.799475    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\" (UniqueName: \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\") pod \"4645fb6e-f9d9-487f-9aa8-920249e2e270\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") "
Oct 30 22:24:42.799708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.799693    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\") pod \"4645fb6e-f9d9-487f-9aa8-920249e2e270\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") "
Oct 30 22:24:42.799770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.799723    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\") pod \"4645fb6e-f9d9-487f-9aa8-920249e2e270\" (UID: \"4645fb6e-f9d9-487f-9aa8-920249e2e270\") "
Oct 30 22:24:42.800108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.800077    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca" pod "4645fb6e-f9d9-487f-9aa8-920249e2e270" (UID: "4645fb6e-f9d9-487f-9aa8-920249e2e270"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:42.801772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.801746    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl" pod "4645fb6e-f9d9-487f-9aa8-920249e2e270" (UID: "4645fb6e-f9d9-487f-9aa8-920249e2e270"). InnerVolumeSpecName "kube-api-access-4bmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:42.801772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.801766    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca" pod "4645fb6e-f9d9-487f-9aa8-920249e2e270" (UID: "4645fb6e-f9d9-487f-9aa8-920249e2e270"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:42.900979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.900940    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:42.900979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.900973    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4645fb6e-f9d9-487f-9aa8-920249e2e270-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:42.901206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:42.900990    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4bmdl\" (UniqueName: \"kubernetes.io/projected/4645fb6e-f9d9-487f-9aa8-920249e2e270-kube-api-access-4bmdl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:43.572933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:43.572902    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"
Oct 30 22:24:43.573128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:43.572940    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz" event={"ID":"4645fb6e-f9d9-487f-9aa8-920249e2e270","Type":"ContainerDied","Data":"07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff"}
Oct 30 22:24:43.573128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:43.573027    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07b94fee45ccff656f7ba92b736c04d42deb7261a8bde15ba992ebbfb0b513ff"
Oct 30 22:24:44.923767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:44.923731    4882 patch_prober.go:29] interesting pod/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:24:44.924196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:44.923789    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:24:46.929379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:46.929317    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:24:47.050328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050298    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050379    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050440    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050494    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050633    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050666    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:47.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050719    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050752    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:47.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050771    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.051043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050814    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\") pod \"69a06f34-f671-49f3-961c-45adef24efac\" (UID: \"69a06f34-f671-49f3-961c-45adef24efac\") "
Oct 30 22:24:47.051043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.050911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:47.051043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051027    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:47.051203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051034    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:47.051343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051325    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.051406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051346    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69a06f34-f671-49f3-961c-45adef24efac-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.051406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051359    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.051406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051374    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.051406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.051385    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.052940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.052913    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:47.053064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.052973    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "kube-api-access-9bgs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:47.053185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.053162    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:24:47.053231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.053188    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo" pod "69a06f34-f671-49f3-961c-45adef24efac" (UID: "69a06f34-f671-49f3-961c-45adef24efac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:24:47.174719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.174573    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bgs4\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-kube-api-access-9bgs4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.174719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.174654    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69a06f34-f671-49f3-961c-45adef24efac-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.174719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.174670    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69a06f34-f671-49f3-961c-45adef24efac-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.174719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.174683    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69a06f34-f671-49f3-961c-45adef24efac-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:47.601828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.601792    4882 generic.go:356] "Generic (PLEG): container finished" podID="69a06f34-f671-49f3-961c-45adef24efac" containerID="6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898" exitCode=0
Oct 30 22:24:47.602021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.601857    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"
Oct 30 22:24:47.602021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.601886    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" event={"ID":"69a06f34-f671-49f3-961c-45adef24efac","Type":"ContainerDied","Data":"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"}
Oct 30 22:24:47.602021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.601951    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw" event={"ID":"69a06f34-f671-49f3-961c-45adef24efac","Type":"ContainerDied","Data":"6eae4235a3604a20ab6965a28546587458308492a7c8eba726c1f1e3862359ff"}
Oct 30 22:24:47.602021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.601982    4882 scope.go:122] "RemoveContainer" containerID="6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"
Oct 30 22:24:47.622955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.622921    4882 scope.go:122] "RemoveContainer" containerID="6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"
Oct 30 22:24:47.623429 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:47.623400    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898\": container with ID starting with 6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898 not found: ID does not exist" containerID="6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"
Oct 30 22:24:47.623523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.623445    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898"} err="failed to get container status \"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898\": rpc error: code = NotFound desc = could not find container \"6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898\": container with ID starting with 6821ca0a8c29ec87a547f6a48033747ee7dd8e98e1e258504a220a5278daf898 not found: ID does not exist"
Oct 30 22:24:47.771035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.770996    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:24:47.774767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:47.774742    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-b-85d00c-openshift-default-7699c795d9-7srqw"]
Oct 30 22:24:48.944239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:48.944206    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="69a06f34-f671-49f3-961c-45adef24efac" path="/var/lib/kubelet/pods/69a06f34-f671-49f3-961c-45adef24efac/volumes"
Oct 30 22:24:49.334468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.334434    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"]
Oct 30 22:24:49.334996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.334982    4882 cpu_manager.go:414] "containerMap: removing container" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy"
Oct 30 22:24:49.335042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.334998    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy"
Oct 30 22:24:49.335083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.335047    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4645fb6e-f9d9-487f-9aa8-920249e2e270" containerName="revoke-keys"
Oct 30 22:24:49.335083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.335052    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4645fb6e-f9d9-487f-9aa8-920249e2e270" containerName="revoke-keys"
Oct 30 22:24:49.335150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.335135    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4645fb6e-f9d9-487f-9aa8-920249e2e270" containerName="revoke-keys"
Oct 30 22:24:49.335150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.335145    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="69a06f34-f671-49f3-961c-45adef24efac" containerName="istio-proxy"
Oct 30 22:24:49.339977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.339954    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.393206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.393173    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"]
Oct 30 22:24:49.395224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.395192    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pwp5\" (UniqueName: \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.395365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.395249    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.395442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.395382    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.439865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.439829    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:24:49.439865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.439841    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:24:49.496279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.496243    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.496453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.496378    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pwp5\" (UniqueName: \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.496453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.496429    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.497100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.497081    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.498730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.498706    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.558764 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.558733    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pwp5\" (UniqueName: \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\") pod \"maas-api-revoke-sub-0a2cf186-dfbbq\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:49.654612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:49.654493    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:50.033690 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:24:50.033647    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fb113cc_25c2_44d0_afb9_db0c491b3897.slice/crio-9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100 WatchSource:0}: Error finding container 9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100: Status 404 returned error can't find the container with id 9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100
Oct 30 22:24:50.037484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.037457    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"]
Oct 30 22:24:50.625874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.625765    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq" event={"ID":"7fb113cc-25c2-44d0-afb9-db0c491b3897","Type":"ContainerStarted","Data":"2730b2b0ea33d87ec938063497edf91ff6a35e2b6d8cdbfa1922570215023577"}
Oct 30 22:24:50.625874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.625828    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq" event={"ID":"7fb113cc-25c2-44d0-afb9-db0c491b3897","Type":"ContainerStarted","Data":"9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100"}
Oct 30 22:24:50.760106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.760072    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:24:50.781799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.781743    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq" podStartSLOduration=1.781727744 podStartE2EDuration="1.781727744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:24:50.778947167 +0000 UTC m=+2813.952271596" watchObservedRunningTime="2026-10-30 22:24:50.781727744 +0000 UTC m=+2813.955052171"
Oct 30 22:24:50.808334 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:50.808149    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:24:50.879292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.879202    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:24:50.983286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:50.983253    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:24:51.635694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:51.635643    4882 generic.go:356] "Generic (PLEG): container finished" podID="7fb113cc-25c2-44d0-afb9-db0c491b3897" containerID="2730b2b0ea33d87ec938063497edf91ff6a35e2b6d8cdbfa1922570215023577" exitCode=0
Oct 30 22:24:51.636263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:51.635696    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq" event={"ID":"7fb113cc-25c2-44d0-afb9-db0c491b3897","Type":"ContainerDied","Data":"2730b2b0ea33d87ec938063497edf91ff6a35e2b6d8cdbfa1922570215023577"}
Oct 30 22:24:52.808372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.808342    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:52.835890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.835852    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\" (UniqueName: \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\") pod \"7fb113cc-25c2-44d0-afb9-db0c491b3897\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") "
Oct 30 22:24:52.836087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.835942    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\") pod \"7fb113cc-25c2-44d0-afb9-db0c491b3897\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") "
Oct 30 22:24:52.836087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.835993    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\") pod \"7fb113cc-25c2-44d0-afb9-db0c491b3897\" (UID: \"7fb113cc-25c2-44d0-afb9-db0c491b3897\") "
Oct 30 22:24:52.836300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.836273    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca" pod "7fb113cc-25c2-44d0-afb9-db0c491b3897" (UID: "7fb113cc-25c2-44d0-afb9-db0c491b3897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:52.836502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.836475    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:52.838105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.838085    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5" pod "7fb113cc-25c2-44d0-afb9-db0c491b3897" (UID: "7fb113cc-25c2-44d0-afb9-db0c491b3897"). InnerVolumeSpecName "kube-api-access-4pwp5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:52.838216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.838141    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca" pod "7fb113cc-25c2-44d0-afb9-db0c491b3897" (UID: "7fb113cc-25c2-44d0-afb9-db0c491b3897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:24:52.937544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.937461    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pwp5\" (UniqueName: \"kubernetes.io/projected/7fb113cc-25c2-44d0-afb9-db0c491b3897-kube-api-access-4pwp5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:52.937544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:52.937499    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7fb113cc-25c2-44d0-afb9-db0c491b3897-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:53.624807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:53.624762    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"]
Oct 30 22:24:53.628303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:53.628272    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-ca6d0b-k4mrb"]
Oct 30 22:24:53.654426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:53.654385    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq" event={"ID":"7fb113cc-25c2-44d0-afb9-db0c491b3897","Type":"ContainerDied","Data":"9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100"}
Oct 30 22:24:53.654426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:53.654407    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"
Oct 30 22:24:53.654634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:53.654435    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9681a89c5503a7b4e2b0c016b36620702885a36908aebade69322291a4ffd100"
Oct 30 22:24:54.943226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:54.943184    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f3cad3b-23c2-4090-a6cb-e4232e0fdc09" path="/var/lib/kubelet/pods/0f3cad3b-23c2-4090-a6cb-e4232e0fdc09/volumes"
Oct 30 22:24:55.880030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:55.879960    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" podUID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" containerName="payload-processing" containerID="cri-o://e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69" gracePeriod=25
Oct 30 22:24:55.984380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:55.984337    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" podUID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" containerName="payload-pre-processing" containerID="cri-o://7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7" gracePeriod=25
Oct 30 22:24:56.165833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.163795    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:24:56.268392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.268364    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:24:56.276532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.276510    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\" (UniqueName: \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\") pod \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") "
Oct 30 22:24:56.276647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.276576    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\") pod \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\" (UID: \"55516d86-a1f5-4369-aaeb-5170f0e2ebb7\") "
Oct 30 22:24:56.276934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.276911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume" pod "55516d86-a1f5-4369-aaeb-5170f0e2ebb7" (UID: "55516d86-a1f5-4369-aaeb-5170f0e2ebb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:56.279016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.278989    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl" pod "55516d86-a1f5-4369-aaeb-5170f0e2ebb7" (UID: "55516d86-a1f5-4369-aaeb-5170f0e2ebb7"). InnerVolumeSpecName "kube-api-access-kr5rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:56.378061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378025    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\") pod \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") "
Oct 30 22:24:56.378261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378117    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\" (UniqueName: \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\") pod \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\" (UID: \"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb\") "
Oct 30 22:24:56.378518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378489    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume" pod "e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" (UID: "e4f19f7a-cad6-4be1-b6ff-bd124c8acadb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:24:56.378700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378683    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kr5rl\" (UniqueName: \"kubernetes.io/projected/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-kube-api-access-kr5rl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.378753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378703    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.378753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.378716    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/55516d86-a1f5-4369-aaeb-5170f0e2ebb7-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.380299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.380276    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf" pod "e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" (UID: "e4f19f7a-cad6-4be1-b6ff-bd124c8acadb"). InnerVolumeSpecName "kube-api-access-z6pdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:24:56.480393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.480315    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6pdf\" (UniqueName: \"kubernetes.io/projected/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb-kube-api-access-z6pdf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:24:56.674580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.674531    4882 generic.go:356] "Generic (PLEG): container finished" podID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" containerID="7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7" exitCode=0
Oct 30 22:24:56.674580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.674580    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"
Oct 30 22:24:56.674855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.674628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" event={"ID":"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb","Type":"ContainerDied","Data":"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"}
Oct 30 22:24:56.674855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.674774    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv" event={"ID":"e4f19f7a-cad6-4be1-b6ff-bd124c8acadb","Type":"ContainerDied","Data":"cdb0c90620fd85ac2940b825a0654e03256a9d26e47a7b5ed6ac3f9c2f968932"}
Oct 30 22:24:56.674855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.674820    4882 scope.go:122] "RemoveContainer" containerID="7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"
Oct 30 22:24:56.676320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.676299    4882 generic.go:356] "Generic (PLEG): container finished" podID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" containerID="e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69" exitCode=0
Oct 30 22:24:56.676410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.676346    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"
Oct 30 22:24:56.676410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.676390    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" event={"ID":"55516d86-a1f5-4369-aaeb-5170f0e2ebb7","Type":"ContainerDied","Data":"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"}
Oct 30 22:24:56.676499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.676432    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4" event={"ID":"55516d86-a1f5-4369-aaeb-5170f0e2ebb7","Type":"ContainerDied","Data":"7b35835737bdbc71f15bfb4408fd55c2869dc9aa710ab4635345085f292822b6"}
Oct 30 22:24:56.685843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.685721    4882 scope.go:122] "RemoveContainer" containerID="7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"
Oct 30 22:24:56.686133 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:56.686113    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7\": container with ID starting with 7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7 not found: ID does not exist" containerID="7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"
Oct 30 22:24:56.686198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.686141    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7"} err="failed to get container status \"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7\": rpc error: code = NotFound desc = could not find container \"7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7\": container with ID starting with 7d5e0e93ae7d6c7e3797325c6a9ac32a7c734e21049b7e6bf1b6f6e4d63d9bd7 not found: ID does not exist"
Oct 30 22:24:56.686198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.686157    4882 scope.go:122] "RemoveContainer" containerID="e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"
Oct 30 22:24:56.695419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.695400    4882 scope.go:122] "RemoveContainer" containerID="e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"
Oct 30 22:24:56.695690 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:24:56.695673    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69\": container with ID starting with e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69 not found: ID does not exist" containerID="e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"
Oct 30 22:24:56.695754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.695698    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69"} err="failed to get container status \"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69\": rpc error: code = NotFound desc = could not find container \"e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69\": container with ID starting with e34b69f82459fc0359b7b4ae8fc6270ecb824b7a36c2e8c5387ff26170c5aa69 not found: ID does not exist"
Oct 30 22:24:56.843394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.843359    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:24:56.893637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.893606    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-a-f4a424-7d85fbf855-nqncv"]
Oct 30 22:24:56.943554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.943517    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" path="/var/lib/kubelet/pods/e4f19f7a-cad6-4be1-b6ff-bd124c8acadb/volumes"
Oct 30 22:24:56.971355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.971152    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:24:56.975304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:56.975276    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f4a424-59c8b96697-gs4g4"]
Oct 30 22:24:58.895522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.895486    4882 scope.go:122] "RemoveContainer" containerID="995776d5ab07648bee126574d3a2b6d45be6e7c3800339bf318285d7ef2d9c8b"
Oct 30 22:24:58.905567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.905545    4882 scope.go:122] "RemoveContainer" containerID="d22d42091da4ed506017c53fb106338fd8111866666a87064fae6c5ffef732c1"
Oct 30 22:24:58.916245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.916218    4882 scope.go:122] "RemoveContainer" containerID="ecf24ddeb1b615025008e758ffc8c9761d79b5a02d72ce5a52ed29caeb69d8f5"
Oct 30 22:24:58.927091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.927067    4882 scope.go:122] "RemoveContainer" containerID="920fc4f6023cb91c5d0f01775bb8ab5eb1ff72a3ad551240196d108de3724f62"
Oct 30 22:24:58.936550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.936523    4882 scope.go:122] "RemoveContainer" containerID="2a5d79dc589070edf3b5938fb6297d39cb0b5d42fcbab2aebbe28084c12aff8e"
Oct 30 22:24:58.944058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:58.944021    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" path="/var/lib/kubelet/pods/55516d86-a1f5-4369-aaeb-5170f0e2ebb7/volumes"
Oct 30 22:24:59.948508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.948464    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:24:59.949900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.949870    4882 cpu_manager.go:414] "containerMap: removing container" podUID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" containerName="payload-processing"
Oct 30 22:24:59.949900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.949899    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" containerName="payload-processing"
Oct 30 22:24:59.950098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.949965    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7fb113cc-25c2-44d0-afb9-db0c491b3897" containerName="revoke-keys"
Oct 30 22:24:59.950098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.949974    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7fb113cc-25c2-44d0-afb9-db0c491b3897" containerName="revoke-keys"
Oct 30 22:24:59.950098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.950004    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" containerName="payload-pre-processing"
Oct 30 22:24:59.950098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.950012    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" containerName="payload-pre-processing"
Oct 30 22:24:59.950299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.950283    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="55516d86-a1f5-4369-aaeb-5170f0e2ebb7" containerName="payload-processing"
Oct 30 22:24:59.950350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.950314    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7fb113cc-25c2-44d0-afb9-db0c491b3897" containerName="revoke-keys"
Oct 30 22:24:59.950350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.950333    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4f19f7a-cad6-4be1-b6ff-bd124c8acadb" containerName="payload-pre-processing"
Oct 30 22:24:59.958220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:24:59.958192    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.007315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.007277    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:25:00.021028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.020996    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.021201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.021049    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.021201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.021171    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.021292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.021214    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.021292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.021255    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.021292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.021282    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvpn6\" (UniqueName: \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.054238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.054208    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f7c488\"/\"praxis-route-f7c488-kserve-self-signed-certs\""
Oct 30 22:25:00.054403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.054262    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-praxis-a-f7c488\"/\"default-dockercfg-6qppf\""
Oct 30 22:25:00.061665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.061637    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f7c488\"/\"openshift-service-ca.crt\""
Oct 30 22:25:00.061767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.061699    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-praxis-a-f7c488\"/\"kube-root-ca.crt\""
Oct 30 22:25:00.122768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.122732    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.122992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.122784    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.122992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.122821    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.122992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.122970    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.123194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.123051    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.123194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.123085    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvpn6\" (UniqueName: \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.123291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.123269    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.123342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.123306    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.123550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.123529    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.125194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.125174    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.125481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.125465    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.185412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.185371    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvpn6\" (UniqueName: \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\") pod \"praxis-route-f7c488-kserve-749d47d9d9-jvrhk\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") " pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.271284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.271183    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:00.650746 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:00.650713    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd865258e_33f3_4161_8413_8887845f19b4.slice/crio-e6c6ba0ec61f0ed07fef812263102c51a6eb8a6bfa97f1f0c07b70a86897c734 WatchSource:0}: Error finding container e6c6ba0ec61f0ed07fef812263102c51a6eb8a6bfa97f1f0c07b70a86897c734: Status 404 returned error can't find the container with id e6c6ba0ec61f0ed07fef812263102c51a6eb8a6bfa97f1f0c07b70a86897c734
Oct 30 22:25:00.653442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.653411    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:25:00.706006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:00.705972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" event={"ID":"d865258e-33f3-4161-8413-8887845f19b4","Type":"ContainerStarted","Data":"e6c6ba0ec61f0ed07fef812263102c51a6eb8a6bfa97f1f0c07b70a86897c734"}
Oct 30 22:25:01.545708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.545673    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"]
Oct 30 22:25:01.561306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.561279    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f4a424-gxsgz"]
Oct 30 22:25:01.713902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.713863    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" event={"ID":"d865258e-33f3-4161-8413-8887845f19b4","Type":"ContainerStarted","Data":"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"}
Oct 30 22:25:01.836502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.836429    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" podStartSLOduration=2.836415613 podStartE2EDuration="2.836415613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:24:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:01.83477026 +0000 UTC m=+2825.008094688" watchObservedRunningTime="2026-10-30 22:25:01.836415613 +0000 UTC m=+2825.009740040"
Oct 30 22:25:01.939839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.939806    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"]
Oct 30 22:25:01.943939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:01.943921    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.003504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.003383    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"]
Oct 30 22:25:02.016707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.016673    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:02.016888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.016737    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:25:02.045637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.045574    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.045845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.045721    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srd52\" (UniqueName: \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.045918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.045863    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.147710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.147614    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.147870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.147729    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.147870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.147774    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srd52\" (UniqueName: \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.148457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.148427    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.150220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.150192    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.210246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.210208    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-srd52\" (UniqueName: \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\") pod \"maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.257924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.257882    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:02.272349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.272320    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:02.632186 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:02.632140    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32ea45aa_36a2_4332_89fb_5095ca0ba49c.slice/crio-e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72 WatchSource:0}: Error finding container e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72: Status 404 returned error can't find the container with id e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72
Oct 30 22:25:02.633774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.633747    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"]
Oct 30 22:25:02.722109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.722062    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542" event={"ID":"32ea45aa-36a2-4332-89fb-5095ca0ba49c","Type":"ContainerStarted","Data":"e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72"}
Oct 30 22:25:02.944467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:02.944441    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4645fb6e-f9d9-487f-9aa8-920249e2e270" path="/var/lib/kubelet/pods/4645fb6e-f9d9-487f-9aa8-920249e2e270/volumes"
Oct 30 22:25:03.730344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:03.730308    4882 generic.go:356] "Generic (PLEG): container finished" podID="32ea45aa-36a2-4332-89fb-5095ca0ba49c" containerID="92deee345eeeed5cd736d2a0e87bb2c112d7bcc0182cbe6426315bca242bdaf4" exitCode=0
Oct 30 22:25:03.730782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:03.730395    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542" event={"ID":"32ea45aa-36a2-4332-89fb-5095ca0ba49c","Type":"ContainerDied","Data":"92deee345eeeed5cd736d2a0e87bb2c112d7bcc0182cbe6426315bca242bdaf4"}
Oct 30 22:25:04.896471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.896441    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:04.981165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.981129    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\") pod \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") "
Oct 30 22:25:04.981338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.981263    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\" (UniqueName: \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\") pod \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") "
Oct 30 22:25:04.981383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.981364    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\") pod \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\" (UID: \"32ea45aa-36a2-4332-89fb-5095ca0ba49c\") "
Oct 30 22:25:04.981790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.981764    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca" pod "32ea45aa-36a2-4332-89fb-5095ca0ba49c" (UID: "32ea45aa-36a2-4332-89fb-5095ca0ba49c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:04.983305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.983279    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca" pod "32ea45aa-36a2-4332-89fb-5095ca0ba49c" (UID: "32ea45aa-36a2-4332-89fb-5095ca0ba49c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:04.983747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:04.983712    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52" pod "32ea45aa-36a2-4332-89fb-5095ca0ba49c" (UID: "32ea45aa-36a2-4332-89fb-5095ca0ba49c"). InnerVolumeSpecName "kube-api-access-srd52". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:05.083098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.083054    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:05.083098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.083099    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32ea45aa-36a2-4332-89fb-5095ca0ba49c-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:05.083098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.083109    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-srd52\" (UniqueName: \"kubernetes.io/projected/32ea45aa-36a2-4332-89fb-5095ca0ba49c-kube-api-access-srd52\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:05.746384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.746354    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"
Oct 30 22:25:05.746553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.746388    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542" event={"ID":"32ea45aa-36a2-4332-89fb-5095ca0ba49c","Type":"ContainerDied","Data":"e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72"}
Oct 30 22:25:05.746553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:05.746458    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e7b447e769d2364b23344a24a21c6c222810b9e9f515a928e2518457c62e4b72"
Oct 30 22:25:07.699618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.699561    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"]
Oct 30 22:25:07.700140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.700126    4882 cpu_manager.go:414] "containerMap: removing container" podUID="32ea45aa-36a2-4332-89fb-5095ca0ba49c" containerName="revoke-keys"
Oct 30 22:25:07.700185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.700140    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32ea45aa-36a2-4332-89fb-5095ca0ba49c" containerName="revoke-keys"
Oct 30 22:25:07.700265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.700248    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="32ea45aa-36a2-4332-89fb-5095ca0ba49c" containerName="revoke-keys"
Oct 30 22:25:07.703779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.703761    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.760780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.760736    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"]
Oct 30 22:25:07.805257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.805216    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:07.805426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.805304    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:25:07.812377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.812341    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nbqmd\" (UniqueName: \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.812532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.812490    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.812641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.812626    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.914134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.914095    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.914337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.914192    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nbqmd\" (UniqueName: \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.914337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.914292    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.915554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.915524    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.922855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.922828    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:07.976079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:07.975999    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nbqmd\" (UniqueName: \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\") pod \"maas-api-revoke-sub-a851f667-rmnrf\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:08.017032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:08.017003    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:08.401462 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:08.401427    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod163d0767_b2d5_43f6_a29d_ee9e38991da1.slice/crio-f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756 WatchSource:0}: Error finding container f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756: Status 404 returned error can't find the container with id f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756
Oct 30 22:25:08.403680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:08.403651    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"]
Oct 30 22:25:08.770956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:08.770915    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf" event={"ID":"163d0767-b2d5-43f6-a29d-ee9e38991da1","Type":"ContainerStarted","Data":"6d812f67dc3657c36b7174b4266e0d2174906d332cad481ce65e9a9c80bd3330"}
Oct 30 22:25:08.771435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:08.770982    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf" event={"ID":"163d0767-b2d5-43f6-a29d-ee9e38991da1","Type":"ContainerStarted","Data":"f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756"}
Oct 30 22:25:08.890242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:08.890183    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf" podStartSLOduration=1.89016834 podStartE2EDuration="1.89016834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:08.889007451 +0000 UTC m=+2832.062331875" watchObservedRunningTime="2026-10-30 22:25:08.89016834 +0000 UTC m=+2832.063492768"
Oct 30 22:25:09.779640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:09.779603    4882 generic.go:356] "Generic (PLEG): container finished" podID="163d0767-b2d5-43f6-a29d-ee9e38991da1" containerID="6d812f67dc3657c36b7174b4266e0d2174906d332cad481ce65e9a9c80bd3330" exitCode=0
Oct 30 22:25:09.780109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:09.779677    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf" event={"ID":"163d0767-b2d5-43f6-a29d-ee9e38991da1","Type":"ContainerDied","Data":"6d812f67dc3657c36b7174b4266e0d2174906d332cad481ce65e9a9c80bd3330"}
Oct 30 22:25:10.271581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:10.271540    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:10.284396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:10.284358    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:10.798499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:10.798151    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:25:10.943868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:10.943839    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:11.049777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.049692    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\" (UniqueName: \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\") pod \"163d0767-b2d5-43f6-a29d-ee9e38991da1\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") "
Oct 30 22:25:11.049961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.049798    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\") pod \"163d0767-b2d5-43f6-a29d-ee9e38991da1\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") "
Oct 30 22:25:11.049961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.049873    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\") pod \"163d0767-b2d5-43f6-a29d-ee9e38991da1\" (UID: \"163d0767-b2d5-43f6-a29d-ee9e38991da1\") "
Oct 30 22:25:11.050243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.050212    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca" pod "163d0767-b2d5-43f6-a29d-ee9e38991da1" (UID: "163d0767-b2d5-43f6-a29d-ee9e38991da1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:11.050735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.050711    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:11.052162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.052137    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca" pod "163d0767-b2d5-43f6-a29d-ee9e38991da1" (UID: "163d0767-b2d5-43f6-a29d-ee9e38991da1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:11.052297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.052275    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd" pod "163d0767-b2d5-43f6-a29d-ee9e38991da1" (UID: "163d0767-b2d5-43f6-a29d-ee9e38991da1"). InnerVolumeSpecName "kube-api-access-nbqmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:11.152124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.152089    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nbqmd\" (UniqueName: \"kubernetes.io/projected/163d0767-b2d5-43f6-a29d-ee9e38991da1-kube-api-access-nbqmd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:11.152124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.152133    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/163d0767-b2d5-43f6-a29d-ee9e38991da1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:11.794009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.793971    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"
Oct 30 22:25:11.794009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.793978    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf" event={"ID":"163d0767-b2d5-43f6-a29d-ee9e38991da1","Type":"ContainerDied","Data":"f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756"}
Oct 30 22:25:11.794231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:11.794030    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f6b0588806f7304b42746f44a8fac121e3d0e2bb635aef3c2979cdebf65cc756"
Oct 30 22:25:13.142662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:13.142624    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:25:18.143223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.143152    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerName="payload-pre-processing" containerID="cri-o://95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c" gracePeriod=25
Oct 30 22:25:18.274569 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:18.274535    4882 logging.go:55] [core] [Channel #3029 SubChannel #3030]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.0.208:50052", ServerName: "10.132.0.208:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.0.208:50052: connect: connection refused"
Oct 30 22:25:18.420113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.420084    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:25:18.534374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.534344    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\") pod \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") "
Oct 30 22:25:18.534614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.534409    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\" (UniqueName: \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\") pod \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\" (UID: \"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b\") "
Oct 30 22:25:18.534866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.534837    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume" pod "1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" (UID: "1ea19cbf-d964-405b-bfb5-c8ca8ed4573b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:18.537278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.536990    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c" pod "1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" (UID: "1ea19cbf-d964-405b-bfb5-c8ca8ed4573b"). InnerVolumeSpecName "kube-api-access-mtv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:18.635921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.635883    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:18.635921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.635916    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtv8c\" (UniqueName: \"kubernetes.io/projected/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b-kube-api-access-mtv8c\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:18.839690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.839658    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerID="95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c" exitCode=0
Oct 30 22:25:18.839855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.839705    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"
Oct 30 22:25:18.839855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.839741    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" event={"ID":"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b","Type":"ContainerDied","Data":"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"}
Oct 30 22:25:18.839855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.839818    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" event={"ID":"1ea19cbf-d964-405b-bfb5-c8ca8ed4573b","Type":"ContainerDied","Data":"1fba8210c84649f05978bbcd42a6842b282a5453486f867143ed773d688cfe35"}
Oct 30 22:25:18.839855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.839840    4882 scope.go:122] "RemoveContainer" containerID="95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"
Oct 30 22:25:18.850649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.850627    4882 scope.go:122] "RemoveContainer" containerID="95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"
Oct 30 22:25:18.850933 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:25:18.850915    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c\": container with ID starting with 95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c not found: ID does not exist" containerID="95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"
Oct 30 22:25:18.850976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:18.850943    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c"} err="failed to get container status \"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c\": rpc error: code = NotFound desc = could not find container \"95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c\": container with ID starting with 95e454036865d246efa10e3fea19ff2f4975879eee01fc826901e742decabb9c not found: ID does not exist"
Oct 30 22:25:19.007304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.007271    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:25:19.057437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.057403    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4"]
Oct 30 22:25:19.167541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.167463    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"]
Oct 30 22:25:19.168006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.167992    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerName="payload-pre-processing"
Oct 30 22:25:19.168053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.168007    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerName="payload-pre-processing"
Oct 30 22:25:19.168053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.168024    4882 cpu_manager.go:414] "containerMap: removing container" podUID="163d0767-b2d5-43f6-a29d-ee9e38991da1" containerName="revoke-keys"
Oct 30 22:25:19.168053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.168031    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="163d0767-b2d5-43f6-a29d-ee9e38991da1" containerName="revoke-keys"
Oct 30 22:25:19.168145 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.168136    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="163d0767-b2d5-43f6-a29d-ee9e38991da1" containerName="revoke-keys"
Oct 30 22:25:19.168178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.168153    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerName="payload-pre-processing"
Oct 30 22:25:19.172973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.172948    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.226031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.225990    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"]
Oct 30 22:25:19.226289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.226253    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:19.226383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.226287    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:25:19.274965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.274920    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-fc3918-6f76c98cdc-gcdk4" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.0.208:50052\" within 1s: context deadline exceeded"
Oct 30 22:25:19.343143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.343099    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.343321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.343221    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.343321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.343247    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7mgv\" (UniqueName: \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.444988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.444894    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.444988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.444950    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.444988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.444971    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7mgv\" (UniqueName: \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.445576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.445551    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.447406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.447383    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.506989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.506952    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7mgv\" (UniqueName: \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\") pod \"maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:19.786262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:19.786181    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:20.161048 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:20.161015    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod718e241d_77e3_4c12_aece_5a959a30eb61.slice/crio-95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98 WatchSource:0}: Error finding container 95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98: Status 404 returned error can't find the container with id 95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98
Oct 30 22:25:20.163647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:20.163573    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"]
Oct 30 22:25:20.810101 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:25:20.809992    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:25:20.858180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:20.858143    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw" event={"ID":"718e241d-77e3-4c12-aece-5a959a30eb61","Type":"ContainerStarted","Data":"d9bf1e864d2630c92c8487065cbecd8ff99a9d78ee63a1ceae0790d427ebbf0b"}
Oct 30 22:25:20.858379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:20.858201    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw" event={"ID":"718e241d-77e3-4c12-aece-5a959a30eb61","Type":"ContainerStarted","Data":"95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98"}
Oct 30 22:25:20.942903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:20.942869    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ea19cbf-d964-405b-bfb5-c8ca8ed4573b" path="/var/lib/kubelet/pods/1ea19cbf-d964-405b-bfb5-c8ca8ed4573b/volumes"
Oct 30 22:25:20.977107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:20.977053    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw" podStartSLOduration=1.977037473 podStartE2EDuration="1.977037473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:20.976146107 +0000 UTC m=+2844.149470532" watchObservedRunningTime="2026-10-30 22:25:20.977037473 +0000 UTC m=+2844.150361898"
Oct 30 22:25:21.866692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:21.866659    4882 generic.go:356] "Generic (PLEG): container finished" podID="718e241d-77e3-4c12-aece-5a959a30eb61" containerID="d9bf1e864d2630c92c8487065cbecd8ff99a9d78ee63a1ceae0790d427ebbf0b" exitCode=0
Oct 30 22:25:21.866692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:21.866699    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw" event={"ID":"718e241d-77e3-4c12-aece-5a959a30eb61","Type":"ContainerDied","Data":"d9bf1e864d2630c92c8487065cbecd8ff99a9d78ee63a1ceae0790d427ebbf0b"}
Oct 30 22:25:23.035008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.034978    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:23.087985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.087947    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\") pod \"718e241d-77e3-4c12-aece-5a959a30eb61\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") "
Oct 30 22:25:23.088155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.088039    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\") pod \"718e241d-77e3-4c12-aece-5a959a30eb61\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") "
Oct 30 22:25:23.088155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.088079    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\" (UniqueName: \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\") pod \"718e241d-77e3-4c12-aece-5a959a30eb61\" (UID: \"718e241d-77e3-4c12-aece-5a959a30eb61\") "
Oct 30 22:25:23.088424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.088393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca" pod "718e241d-77e3-4c12-aece-5a959a30eb61" (UID: "718e241d-77e3-4c12-aece-5a959a30eb61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:23.088654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.088620    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.090756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.090733    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca" pod "718e241d-77e3-4c12-aece-5a959a30eb61" (UID: "718e241d-77e3-4c12-aece-5a959a30eb61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:23.090756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.090750    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv" pod "718e241d-77e3-4c12-aece-5a959a30eb61" (UID: "718e241d-77e3-4c12-aece-5a959a30eb61"). InnerVolumeSpecName "kube-api-access-r7mgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:23.190079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.190001    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/718e241d-77e3-4c12-aece-5a959a30eb61-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.190079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.190030    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7mgv\" (UniqueName: \"kubernetes.io/projected/718e241d-77e3-4c12-aece-5a959a30eb61-kube-api-access-r7mgv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:23.880157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.880123    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw" event={"ID":"718e241d-77e3-4c12-aece-5a959a30eb61","Type":"ContainerDied","Data":"95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98"}
Oct 30 22:25:23.880339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.880173    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95b44bad9535e014c5af57a711de2acf12303e9b30c9b4d16603bc9615fdec98"
Oct 30 22:25:23.880339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:23.880130    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"
Oct 30 22:25:24.024920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.024889    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"]
Oct 30 22:25:24.025418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.025404    4882 cpu_manager.go:414] "containerMap: removing container" podUID="718e241d-77e3-4c12-aece-5a959a30eb61" containerName="revoke-keys"
Oct 30 22:25:24.025484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.025418    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="718e241d-77e3-4c12-aece-5a959a30eb61" containerName="revoke-keys"
Oct 30 22:25:24.025539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.025520    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="718e241d-77e3-4c12-aece-5a959a30eb61" containerName="revoke-keys"
Oct 30 22:25:24.029118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.029101    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.082179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.082139    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:24.082599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.082197    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:25:24.083050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.083029    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"]
Oct 30 22:25:24.098940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.098909    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.099107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.099047    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.099152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.099122    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4zdww\" (UniqueName: \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.200527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.200441    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.200527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.200484    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4zdww\" (UniqueName: \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.200776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.200580    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.201281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.201250    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.202949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.202930    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.262720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.262679    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4zdww\" (UniqueName: \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\") pod \"maas-api-revoke-sub-859024a3-jq9ll\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.343253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.343216    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:24.723577 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:24.723542    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf45c3407_3cbe_4e72_b54b_de04a3446838.slice/crio-35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9 WatchSource:0}: Error finding container 35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9: Status 404 returned error can't find the container with id 35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9
Oct 30 22:25:24.725737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.725704    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"]
Oct 30 22:25:24.887973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:24.887935    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll" event={"ID":"f45c3407-3cbe-4e72-b54b-de04a3446838","Type":"ContainerStarted","Data":"35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9"}
Oct 30 22:25:25.895531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:25.895437    4882 generic.go:356] "Generic (PLEG): container finished" podID="f45c3407-3cbe-4e72-b54b-de04a3446838" containerID="dd475929aa8678ad4426c8db17b02119ae1c9e3c6e10f61dffa3336cec8715bb" exitCode=0
Oct 30 22:25:25.895531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:25.895485    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll" event={"ID":"f45c3407-3cbe-4e72-b54b-de04a3446838","Type":"ContainerDied","Data":"dd475929aa8678ad4426c8db17b02119ae1c9e3c6e10f61dffa3336cec8715bb"}
Oct 30 22:25:27.059200 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.059170    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:27.133802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.133769    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\") pod \"f45c3407-3cbe-4e72-b54b-de04a3446838\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") "
Oct 30 22:25:27.134013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.133953    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\" (UniqueName: \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\") pod \"f45c3407-3cbe-4e72-b54b-de04a3446838\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") "
Oct 30 22:25:27.134013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.133998    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\") pod \"f45c3407-3cbe-4e72-b54b-de04a3446838\" (UID: \"f45c3407-3cbe-4e72-b54b-de04a3446838\") "
Oct 30 22:25:27.134194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.134169    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca" pod "f45c3407-3cbe-4e72-b54b-de04a3446838" (UID: "f45c3407-3cbe-4e72-b54b-de04a3446838"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:27.134348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.134327    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.136076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.136048    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww" pod "f45c3407-3cbe-4e72-b54b-de04a3446838" (UID: "f45c3407-3cbe-4e72-b54b-de04a3446838"). InnerVolumeSpecName "kube-api-access-4zdww". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:27.136164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.136125    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca" pod "f45c3407-3cbe-4e72-b54b-de04a3446838" (UID: "f45c3407-3cbe-4e72-b54b-de04a3446838"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:27.235279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.235196    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4zdww\" (UniqueName: \"kubernetes.io/projected/f45c3407-3cbe-4e72-b54b-de04a3446838-kube-api-access-4zdww\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.235279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.235231    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f45c3407-3cbe-4e72-b54b-de04a3446838-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:27.908670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.908639    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"
Oct 30 22:25:27.908670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.908644    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll" event={"ID":"f45c3407-3cbe-4e72-b54b-de04a3446838","Type":"ContainerDied","Data":"35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9"}
Oct 30 22:25:27.908900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:27.908716    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35b213c1fdaca851fc4145e8123a306497d4f8bbcbfbbc640d46c8bdee1b70b9"
Oct 30 22:25:31.566291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:31.566255    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:25:33.620670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.620636    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"]
Oct 30 22:25:33.621290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.621277    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f45c3407-3cbe-4e72-b54b-de04a3446838" containerName="revoke-keys"
Oct 30 22:25:33.621332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.621291    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f45c3407-3cbe-4e72-b54b-de04a3446838" containerName="revoke-keys"
Oct 30 22:25:33.621396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.621388    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f45c3407-3cbe-4e72-b54b-de04a3446838" containerName="revoke-keys"
Oct 30 22:25:33.624827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.624811    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.677885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.677847    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:25:33.679340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.679312    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"]
Oct 30 22:25:33.707147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.707112    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.707147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.707148    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.707392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.707224    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czmrv\" (UniqueName: \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.725391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.725364    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:25:33.808473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.808432    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.808473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.808477    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.808743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.808506    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czmrv\" (UniqueName: \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.809216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.809183    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.810855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.810829    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.870108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.870073    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czmrv\" (UniqueName: \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\") pod \"maas-api-revoke-sub-f83b7654-7v8tz\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:33.938647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:33.938561    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:34.313797 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:25:34.313763    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0aef2d8b_0716_4f7a_b906_899853b1abf0.slice/crio-d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513 WatchSource:0}: Error finding container d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513: Status 404 returned error can't find the container with id d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513
Oct 30 22:25:34.316222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:34.316195    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"]
Oct 30 22:25:34.960786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:34.960698    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz" event={"ID":"0aef2d8b-0716-4f7a-b906-899853b1abf0","Type":"ContainerStarted","Data":"72ccbf4275cce1a9ba41ccf05f440dbe062c3794f7d5415f9cef8f500e4dcc7a"}
Oct 30 22:25:34.960786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:34.960757    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz" event={"ID":"0aef2d8b-0716-4f7a-b906-899853b1abf0","Type":"ContainerStarted","Data":"d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513"}
Oct 30 22:25:35.079734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:35.079684    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz" podStartSLOduration=2.079671431 podStartE2EDuration="2.079671431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:25:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:25:35.077316404 +0000 UTC m=+2858.250640829" watchObservedRunningTime="2026-10-30 22:25:35.079671431 +0000 UTC m=+2858.252995857"
Oct 30 22:25:35.968765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:35.968724    4882 generic.go:356] "Generic (PLEG): container finished" podID="0aef2d8b-0716-4f7a-b906-899853b1abf0" containerID="72ccbf4275cce1a9ba41ccf05f440dbe062c3794f7d5415f9cef8f500e4dcc7a" exitCode=0
Oct 30 22:25:35.969171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:35.968809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz" event={"ID":"0aef2d8b-0716-4f7a-b906-899853b1abf0","Type":"ContainerDied","Data":"72ccbf4275cce1a9ba41ccf05f440dbe062c3794f7d5415f9cef8f500e4dcc7a"}
Oct 30 22:25:36.567074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.567033    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" podUID="ab56c39e-ec82-4409-a289-fe89b6727b5e" containerName="payload-processing" containerID="cri-o://f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f" gracePeriod=25
Oct 30 22:25:36.838281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.838254    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:25:36.949066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.949032    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\") pod \"ab56c39e-ec82-4409-a289-fe89b6727b5e\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") "
Oct 30 22:25:36.949234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.949078    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\" (UniqueName: \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\") pod \"ab56c39e-ec82-4409-a289-fe89b6727b5e\" (UID: \"ab56c39e-ec82-4409-a289-fe89b6727b5e\") "
Oct 30 22:25:36.949389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.949366    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume" pod "ab56c39e-ec82-4409-a289-fe89b6727b5e" (UID: "ab56c39e-ec82-4409-a289-fe89b6727b5e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:36.949581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.949564    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ab56c39e-ec82-4409-a289-fe89b6727b5e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:36.951288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.951263    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw" pod "ab56c39e-ec82-4409-a289-fe89b6727b5e" (UID: "ab56c39e-ec82-4409-a289-fe89b6727b5e"). InnerVolumeSpecName "kube-api-access-rx5kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:36.976455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.976420    4882 generic.go:356] "Generic (PLEG): container finished" podID="ab56c39e-ec82-4409-a289-fe89b6727b5e" containerID="f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f" exitCode=0
Oct 30 22:25:36.976932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.976468    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"
Oct 30 22:25:36.976932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.976496    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" event={"ID":"ab56c39e-ec82-4409-a289-fe89b6727b5e","Type":"ContainerDied","Data":"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"}
Oct 30 22:25:36.976932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.976549    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx" event={"ID":"ab56c39e-ec82-4409-a289-fe89b6727b5e","Type":"ContainerDied","Data":"82d5cfe8a138950dc8c23d1d6a215fc67ec47061ae0eea519423524e786ddb6b"}
Oct 30 22:25:36.976932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.976572    4882 scope.go:122] "RemoveContainer" containerID="f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"
Oct 30 22:25:36.988792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.988761    4882 scope.go:122] "RemoveContainer" containerID="f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"
Oct 30 22:25:36.989157 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:25:36.989131    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f\": container with ID starting with f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f not found: ID does not exist" containerID="f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"
Oct 30 22:25:36.989252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:36.989173    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f"} err="failed to get container status \"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f\": rpc error: code = NotFound desc = could not find container \"f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f\": container with ID starting with f677ffecd3e5ac7bab75bf0f18171faceb5c17347c962774e07d5eddff8abc2f not found: ID does not exist"
Oct 30 22:25:37.050807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.050770    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rx5kw\" (UniqueName: \"kubernetes.io/projected/ab56c39e-ec82-4409-a289-fe89b6727b5e-kube-api-access-rx5kw\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:37.123334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.123300    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:37.184831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.184798    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:25:37.188916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.188888    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-93131d-58db56c5dc-zq5vx"]
Oct 30 22:25:37.252657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.252620    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\") pod \"0aef2d8b-0716-4f7a-b906-899853b1abf0\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") "
Oct 30 22:25:37.252827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.252737    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\") pod \"0aef2d8b-0716-4f7a-b906-899853b1abf0\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") "
Oct 30 22:25:37.252827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.252819    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\" (UniqueName: \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\") pod \"0aef2d8b-0716-4f7a-b906-899853b1abf0\" (UID: \"0aef2d8b-0716-4f7a-b906-899853b1abf0\") "
Oct 30 22:25:37.253085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.253059    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca" pod "0aef2d8b-0716-4f7a-b906-899853b1abf0" (UID: "0aef2d8b-0716-4f7a-b906-899853b1abf0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:25:37.253184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.253170    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:37.254675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.254651    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca" pod "0aef2d8b-0716-4f7a-b906-899853b1abf0" (UID: "0aef2d8b-0716-4f7a-b906-899853b1abf0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:25:37.254836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.254818    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv" pod "0aef2d8b-0716-4f7a-b906-899853b1abf0" (UID: "0aef2d8b-0716-4f7a-b906-899853b1abf0"). InnerVolumeSpecName "kube-api-access-czmrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:25:37.354680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.354576    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0aef2d8b-0716-4f7a-b906-899853b1abf0-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:37.354680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.354634    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czmrv\" (UniqueName: \"kubernetes.io/projected/0aef2d8b-0716-4f7a-b906-899853b1abf0-kube-api-access-czmrv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:25:37.983821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.983791    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"
Oct 30 22:25:37.984275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.983818    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz" event={"ID":"0aef2d8b-0716-4f7a-b906-899853b1abf0","Type":"ContainerDied","Data":"d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513"}
Oct 30 22:25:37.984275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:37.983872    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3be92300d4604251f3d9948a8cb72ba1b0fe475d36f45dfb55e06730ed47513"
Oct 30 22:25:38.943520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:38.943483    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab56c39e-ec82-4409-a289-fe89b6727b5e" path="/var/lib/kubelet/pods/ab56c39e-ec82-4409-a289-fe89b6727b5e/volumes"
Oct 30 22:25:45.245166 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:45.245121    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:25:50.808398 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:25:50.808356    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:25:55.857374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:55.857331    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"]
Oct 30 22:25:55.907856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:55.907819    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-fc3918-mn542"]
Oct 30 22:25:56.943389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:56.943355    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32ea45aa-36a2-4332-89fb-5095ca0ba49c" path="/var/lib/kubelet/pods/32ea45aa-36a2-4332-89fb-5095ca0ba49c/volumes"
Oct 30 22:25:59.013712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:25:59.013682    4882 scope.go:122] "RemoveContainer" containerID="0c8d2b4cd274d90e834611a7720c449e33e4a1e53761ef810db9be34be616193"
Oct 30 22:26:00.132490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.132453    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"]
Oct 30 22:26:00.133066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133051    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0aef2d8b-0716-4f7a-b906-899853b1abf0" containerName="revoke-keys"
Oct 30 22:26:00.133113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133068    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0aef2d8b-0716-4f7a-b906-899853b1abf0" containerName="revoke-keys"
Oct 30 22:26:00.133113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133090    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ab56c39e-ec82-4409-a289-fe89b6727b5e" containerName="payload-processing"
Oct 30 22:26:00.133113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133095    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab56c39e-ec82-4409-a289-fe89b6727b5e" containerName="payload-processing"
Oct 30 22:26:00.133218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133187    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0aef2d8b-0716-4f7a-b906-899853b1abf0" containerName="revoke-keys"
Oct 30 22:26:00.133218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.133199    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab56c39e-ec82-4409-a289-fe89b6727b5e" containerName="payload-processing"
Oct 30 22:26:00.138174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.138154    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.192169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.192131    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"]
Oct 30 22:26:00.215338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.215299    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6s4vc\" (UniqueName: \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.215532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.215349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.215532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.215437    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.238160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.238129    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:00.238370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.238352    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:00.317096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.317058    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6s4vc\" (UniqueName: \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.317300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.317112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.317300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.317142    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.317969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.317937    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.319637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.319613    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.380734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.380693    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6s4vc\" (UniqueName: \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\") pod \"maas-api-revoke-sub-6d6c42d6-zbf58\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.451442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.451362    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:00.827787 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:00.827755    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83385fa7_04ef_4945_bbc0_1a0e6c0f7ec0.slice/crio-9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154 WatchSource:0}: Error finding container 9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154: Status 404 returned error can't find the container with id 9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154
Oct 30 22:26:00.832479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:00.832455    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"]
Oct 30 22:26:01.138510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:01.138478    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58" event={"ID":"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0","Type":"ContainerStarted","Data":"9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154"}
Oct 30 22:26:02.146423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:02.146389    4882 generic.go:356] "Generic (PLEG): container finished" podID="83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" containerID="0e76ee1995c8e3304f940f61aa25094c763f267cdc9969ad85cfb9a37e617de3" exitCode=0
Oct 30 22:26:02.146878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:02.146425    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58" event={"ID":"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0","Type":"ContainerDied","Data":"0e76ee1995c8e3304f940f61aa25094c763f267cdc9969ad85cfb9a37e617de3"}
Oct 30 22:26:03.302896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.302863    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:03.352754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.352720    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\") pod \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") "
Oct 30 22:26:03.352949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.352801    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\" (UniqueName: \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\") pod \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") "
Oct 30 22:26:03.353013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.352988    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\") pod \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\" (UID: \"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0\") "
Oct 30 22:26:03.353171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.353147    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca" pod "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" (UID: "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:03.353647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.353376    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.355059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.355032    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc" pod "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" (UID: "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0"). InnerVolumeSpecName "kube-api-access-6s4vc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:03.355243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.355221    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca" pod "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" (UID: "83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:03.455078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.454982    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:03.455078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:03.455020    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6s4vc\" (UniqueName: \"kubernetes.io/projected/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0-kube-api-access-6s4vc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:04.160906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:04.160865    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58" event={"ID":"83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0","Type":"ContainerDied","Data":"9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154"}
Oct 30 22:26:04.160906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:04.160900    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"
Oct 30 22:26:04.161132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:04.160922    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d1b8f3a9f718ff3e3ae1e1f18bdbff9fe05f625aaeea4fd9c93c484e29ba154"
Oct 30 22:26:11.213309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:11.213216    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"]
Oct 30 22:26:11.263652 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:11.263611    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-93131d-wvnjw"]
Oct 30 22:26:12.943857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:12.943826    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="718e241d-77e3-4c12-aece-5a959a30eb61" path="/var/lib/kubelet/pods/718e241d-77e3-4c12-aece-5a959a30eb61/volumes"
Oct 30 22:26:18.665451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.665414    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"]
Oct 30 22:26:18.666006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.665990    4882 cpu_manager.go:414] "containerMap: removing container" podUID="83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" containerName="revoke-keys"
Oct 30 22:26:18.666062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.666008    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" containerName="revoke-keys"
Oct 30 22:26:18.666148 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.666138    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" containerName="revoke-keys"
Oct 30 22:26:18.670563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.670541    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.724113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.724078    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"]
Oct 30 22:26:18.726690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.726652    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.726875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.726699    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gn4vv\" (UniqueName: \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.726875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.726838    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.771159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.771116    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:18.771357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.771205    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:18.828433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.828386    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.828433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.828429    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gn4vv\" (UniqueName: \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.828750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.828486    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.829131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.829107    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.831134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.831107    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.890600 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.890546    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gn4vv\" (UniqueName: \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\") pod \"maas-api-revoke-sub-7d6988d1-nbfqh\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:18.984338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:18.984246    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:19.365341 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:19.365297    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabd6cdee_2341_43bd_a4b8_7c3d78e95081.slice/crio-07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e WatchSource:0}: Error finding container 07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e: Status 404 returned error can't find the container with id 07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e
Oct 30 22:26:19.368275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:19.368172    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"]
Oct 30 22:26:20.264600 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:20.264550    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh" event={"ID":"abd6cdee-2341-43bd-a4b8-7c3d78e95081","Type":"ContainerStarted","Data":"ddd1c61cfe92858008881587f496c82b82abc10354020ae1a89a34ca9548e133"}
Oct 30 22:26:20.265009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:20.264641    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh" event={"ID":"abd6cdee-2341-43bd-a4b8-7c3d78e95081","Type":"ContainerStarted","Data":"07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e"}
Oct 30 22:26:20.382952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:20.382878    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh" podStartSLOduration=2.382858617 podStartE2EDuration="2.382858617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:20.3821439 +0000 UTC m=+2903.555468327" watchObservedRunningTime="2026-10-30 22:26:20.382858617 +0000 UTC m=+2903.556183045"
Oct 30 22:26:20.809908 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:26:20.809858    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:26:21.273341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:21.273305    4882 generic.go:356] "Generic (PLEG): container finished" podID="abd6cdee-2341-43bd-a4b8-7c3d78e95081" containerID="ddd1c61cfe92858008881587f496c82b82abc10354020ae1a89a34ca9548e133" exitCode=0
Oct 30 22:26:21.273800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:21.273391    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh" event={"ID":"abd6cdee-2341-43bd-a4b8-7c3d78e95081","Type":"ContainerDied","Data":"ddd1c61cfe92858008881587f496c82b82abc10354020ae1a89a34ca9548e133"}
Oct 30 22:26:22.426911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.426879    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:22.473003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.472959    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\") pod \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") "
Oct 30 22:26:22.473213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.473140    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\" (UniqueName: \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\") pod \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") "
Oct 30 22:26:22.473213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.473182    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\") pod \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\" (UID: \"abd6cdee-2341-43bd-a4b8-7c3d78e95081\") "
Oct 30 22:26:22.473342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.473320    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca" pod "abd6cdee-2341-43bd-a4b8-7c3d78e95081" (UID: "abd6cdee-2341-43bd-a4b8-7c3d78e95081"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:22.473580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.473560    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:22.475466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.475439    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca" pod "abd6cdee-2341-43bd-a4b8-7c3d78e95081" (UID: "abd6cdee-2341-43bd-a4b8-7c3d78e95081"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:22.475540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.475485    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv" pod "abd6cdee-2341-43bd-a4b8-7c3d78e95081" (UID: "abd6cdee-2341-43bd-a4b8-7c3d78e95081"). InnerVolumeSpecName "kube-api-access-gn4vv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:22.574838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.574758    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gn4vv\" (UniqueName: \"kubernetes.io/projected/abd6cdee-2341-43bd-a4b8-7c3d78e95081-kube-api-access-gn4vv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:22.574838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:22.574791    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/abd6cdee-2341-43bd-a4b8-7c3d78e95081-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:23.287787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:23.287703    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh" event={"ID":"abd6cdee-2341-43bd-a4b8-7c3d78e95081","Type":"ContainerDied","Data":"07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e"}
Oct 30 22:26:23.287787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:23.287750    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07a4e28dae35f4bec297a2f9ca3d31ee8ee62232a8afc776d245cd97bb3d215e"
Oct 30 22:26:23.287787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:23.287716    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"
Oct 30 22:26:24.867831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:24.867786    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:26:24.868255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:24.868068    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" podUID="36be8729-107f-4459-b3f2-f4f06d6a90a9" containerName="payload-processing" containerID="cri-o://1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873" gracePeriod=30
Oct 30 22:26:24.977487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:24.977450    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:26:24.977724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:24.977701    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" podUID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" containerName="payload-pre-processing" containerID="cri-o://94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010" gracePeriod=30
Oct 30 22:26:25.151807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.151776    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:26:25.204402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.204369    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\") pod \"36be8729-107f-4459-b3f2-f4f06d6a90a9\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") "
Oct 30 22:26:25.204619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.204412    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\" (UniqueName: \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\") pod \"36be8729-107f-4459-b3f2-f4f06d6a90a9\" (UID: \"36be8729-107f-4459-b3f2-f4f06d6a90a9\") "
Oct 30 22:26:25.204952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.204919    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume" pod "36be8729-107f-4459-b3f2-f4f06d6a90a9" (UID: "36be8729-107f-4459-b3f2-f4f06d6a90a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:25.206871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.206836    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q" pod "36be8729-107f-4459-b3f2-f4f06d6a90a9" (UID: "36be8729-107f-4459-b3f2-f4f06d6a90a9"). InnerVolumeSpecName "kube-api-access-gg84q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:25.250130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.250102    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:26:25.304570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.304522    4882 generic.go:356] "Generic (PLEG): container finished" podID="36be8729-107f-4459-b3f2-f4f06d6a90a9" containerID="1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873" exitCode=0
Oct 30 22:26:25.304806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.304607    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" event={"ID":"36be8729-107f-4459-b3f2-f4f06d6a90a9","Type":"ContainerDied","Data":"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"}
Oct 30 22:26:25.304806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.304681    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml" event={"ID":"36be8729-107f-4459-b3f2-f4f06d6a90a9","Type":"ContainerDied","Data":"903a41a6e77ba884f385e1e43e3f3b7b0e724523aa7cc9588d358e3259700dab"}
Oct 30 22:26:25.304806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.304710    4882 scope.go:122] "RemoveContainer" containerID="1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"
Oct 30 22:26:25.304806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.304611    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"
Oct 30 22:26:25.305919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.305286    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\" (UniqueName: \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\") pod \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") "
Oct 30 22:26:25.305919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.305489    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\") pod \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\" (UID: \"c6179c80-1db2-4877-9abe-9f6aaffe97f1\") "
Oct 30 22:26:25.305919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.305865    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/36be8729-107f-4459-b3f2-f4f06d6a90a9-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:25.305919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.305894    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg84q\" (UniqueName: \"kubernetes.io/projected/36be8729-107f-4459-b3f2-f4f06d6a90a9-kube-api-access-gg84q\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:25.306240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.305979    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume" pod "c6179c80-1db2-4877-9abe-9f6aaffe97f1" (UID: "c6179c80-1db2-4877-9abe-9f6aaffe97f1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:25.306894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.306864    4882 generic.go:356] "Generic (PLEG): container finished" podID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" containerID="94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010" exitCode=0
Oct 30 22:26:25.306978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.306922    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" event={"ID":"c6179c80-1db2-4877-9abe-9f6aaffe97f1","Type":"ContainerDied","Data":"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"}
Oct 30 22:26:25.306978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.306970    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9" event={"ID":"c6179c80-1db2-4877-9abe-9f6aaffe97f1","Type":"ContainerDied","Data":"be71a2f658319c9228bd27e9279902eb4d429ce9c4aa68745ff719c03c59d087"}
Oct 30 22:26:25.307093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.306988    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"
Oct 30 22:26:25.312407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.312373    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2" pod "c6179c80-1db2-4877-9abe-9f6aaffe97f1" (UID: "c6179c80-1db2-4877-9abe-9f6aaffe97f1"). InnerVolumeSpecName "kube-api-access-pbdl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:25.316561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.316542    4882 scope.go:122] "RemoveContainer" containerID="1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"
Oct 30 22:26:25.317039 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:26:25.317017    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873\": container with ID starting with 1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873 not found: ID does not exist" containerID="1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"
Oct 30 22:26:25.317119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.317048    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873"} err="failed to get container status \"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873\": rpc error: code = NotFound desc = could not find container \"1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873\": container with ID starting with 1db9eb10b82b885d09ced2d6f28df2fb3f8403e8287a10fe373e8d645722c873 not found: ID does not exist"
Oct 30 22:26:25.317119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.317067    4882 scope.go:122] "RemoveContainer" containerID="94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"
Oct 30 22:26:25.326914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.326889    4882 scope.go:122] "RemoveContainer" containerID="94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"
Oct 30 22:26:25.327194 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:26:25.327175    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010\": container with ID starting with 94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010 not found: ID does not exist" containerID="94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"
Oct 30 22:26:25.327267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.327201    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010"} err="failed to get container status \"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010\": rpc error: code = NotFound desc = could not find container \"94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010\": container with ID starting with 94a1ed9e5a8b90dedf8a8dca69e04fe58b5fb594ee671f97a66f1789613cf010 not found: ID does not exist"
Oct 30 22:26:25.407243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.407167    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c6179c80-1db2-4877-9abe-9f6aaffe97f1-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:25.407243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.407195    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbdl2\" (UniqueName: \"kubernetes.io/projected/c6179c80-1db2-4877-9abe-9f6aaffe97f1-kube-api-access-pbdl2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:25.503225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.503193    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:26:25.507139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.507111    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-63a282-6fd5799d6b-5swml"]
Oct 30 22:26:25.729969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.729873    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:26:25.734498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:25.734464    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-63a282-85bbb794f9-rrrl9"]
Oct 30 22:26:26.579311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:26.579277    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"]
Oct 30 22:26:26.582499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:26.582472    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-63a282-8jqlw"]
Oct 30 22:26:26.944167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:26.944128    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36be8729-107f-4459-b3f2-f4f06d6a90a9" path="/var/lib/kubelet/pods/36be8729-107f-4459-b3f2-f4f06d6a90a9/volumes"
Oct 30 22:26:26.944618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:26.944602    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28" path="/var/lib/kubelet/pods/660b7eb8-0cb8-4c37-8fbc-85ea7bdf8f28/volumes"
Oct 30 22:26:26.944983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:26.944970    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" path="/var/lib/kubelet/pods/c6179c80-1db2-4877-9abe-9f6aaffe97f1/volumes"
Oct 30 22:26:29.479757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.479718    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"]
Oct 30 22:26:29.480226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480213    4882 cpu_manager.go:414] "containerMap: removing container" podUID="abd6cdee-2341-43bd-a4b8-7c3d78e95081" containerName="revoke-keys"
Oct 30 22:26:29.480276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480227    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abd6cdee-2341-43bd-a4b8-7c3d78e95081" containerName="revoke-keys"
Oct 30 22:26:29.480276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480266    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" containerName="payload-pre-processing"
Oct 30 22:26:29.480276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480272    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" containerName="payload-pre-processing"
Oct 30 22:26:29.480373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480292    4882 cpu_manager.go:414] "containerMap: removing container" podUID="36be8729-107f-4459-b3f2-f4f06d6a90a9" containerName="payload-processing"
Oct 30 22:26:29.480373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480297    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36be8729-107f-4459-b3f2-f4f06d6a90a9" containerName="payload-processing"
Oct 30 22:26:29.480441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480382    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6179c80-1db2-4877-9abe-9f6aaffe97f1" containerName="payload-pre-processing"
Oct 30 22:26:29.480441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480395    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="36be8729-107f-4459-b3f2-f4f06d6a90a9" containerName="payload-processing"
Oct 30 22:26:29.480441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.480406    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="abd6cdee-2341-43bd-a4b8-7c3d78e95081" containerName="revoke-keys"
Oct 30 22:26:29.485233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.485214    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.539608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.539549    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:29.554322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.554286    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.554322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.554327    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtbq4\" (UniqueName: \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.554607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.554484    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.589934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.589900    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:29.593161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.593135    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"]
Oct 30 22:26:29.655493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.655447    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.655735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.655498    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtbq4\" (UniqueName: \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.655735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.655632    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.656246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.656226    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.657850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.657826    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.717458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.717412    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtbq4\" (UniqueName: \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\") pod \"maas-api-revoke-sub-0d905bb9-ch4m7\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:29.799199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:29.799113    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:30.175208 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:30.175174    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a2e3d8f_dffb_4a99_b302_da2af97beb4c.slice/crio-195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a WatchSource:0}: Error finding container 195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a: Status 404 returned error can't find the container with id 195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a
Oct 30 22:26:30.178248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:30.178170    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"]
Oct 30 22:26:30.343332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:30.343295    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7" event={"ID":"1a2e3d8f-dffb-4a99-b302-da2af97beb4c","Type":"ContainerStarted","Data":"195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a"}
Oct 30 22:26:31.352043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:31.352006    4882 generic.go:356] "Generic (PLEG): container finished" podID="1a2e3d8f-dffb-4a99-b302-da2af97beb4c" containerID="bc531d4ac2b989532ad2e7c5c54baa4c8189cb22bd6affa2b57959e80541378b" exitCode=0
Oct 30 22:26:31.352467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:31.352094    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7" event={"ID":"1a2e3d8f-dffb-4a99-b302-da2af97beb4c","Type":"ContainerDied","Data":"bc531d4ac2b989532ad2e7c5c54baa4c8189cb22bd6affa2b57959e80541378b"}
Oct 30 22:26:31.993530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:31.993486    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:26:32.499769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.499740    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:32.586351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.586318    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\" (UniqueName: \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\") pod \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") "
Oct 30 22:26:32.586561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.586359    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\") pod \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") "
Oct 30 22:26:32.586677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.586560    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\") pod \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\" (UID: \"1a2e3d8f-dffb-4a99-b302-da2af97beb4c\") "
Oct 30 22:26:32.587000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.586948    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca" pod "1a2e3d8f-dffb-4a99-b302-da2af97beb4c" (UID: "1a2e3d8f-dffb-4a99-b302-da2af97beb4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:32.588682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.588656    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca" pod "1a2e3d8f-dffb-4a99-b302-da2af97beb4c" (UID: "1a2e3d8f-dffb-4a99-b302-da2af97beb4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:32.588799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.588777    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4" pod "1a2e3d8f-dffb-4a99-b302-da2af97beb4c" (UID: "1a2e3d8f-dffb-4a99-b302-da2af97beb4c"). InnerVolumeSpecName "kube-api-access-wtbq4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:32.688131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.688035    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtbq4\" (UniqueName: \"kubernetes.io/projected/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-kube-api-access-wtbq4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:32.688131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.688068    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:32.688131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:32.688082    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1a2e3d8f-dffb-4a99-b302-da2af97beb4c-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:33.366582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:33.366551    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"
Oct 30 22:26:33.366582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:33.366551    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7" event={"ID":"1a2e3d8f-dffb-4a99-b302-da2af97beb4c","Type":"ContainerDied","Data":"195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a"}
Oct 30 22:26:33.366830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:33.366638    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="195473d854fceb3befc0c30ea163c8626e8221af2e677aedd37ddae898d3fc6a"
Oct 30 22:26:35.081573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.081540    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"]
Oct 30 22:26:35.082281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.082257    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1a2e3d8f-dffb-4a99-b302-da2af97beb4c" containerName="revoke-keys"
Oct 30 22:26:35.082380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.082281    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a2e3d8f-dffb-4a99-b302-da2af97beb4c" containerName="revoke-keys"
Oct 30 22:26:35.082500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.082486    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a2e3d8f-dffb-4a99-b302-da2af97beb4c" containerName="revoke-keys"
Oct 30 22:26:35.086045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.086022    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.141183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.141146    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"]
Oct 30 22:26:35.187459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.187423    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:35.187694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.187481    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:35.215406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.215368    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54h7x\" (UniqueName: \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.215627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.215517    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.215687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.215626    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.316245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.316214    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.316462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.316264    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.316462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.316332    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54h7x\" (UniqueName: \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.317137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.317109    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.318711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.318692    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.379203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.379118    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-54h7x\" (UniqueName: \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\") pod \"maas-api-revoke-sub-fa89ed4e-skmvf\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.400129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.400096    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:35.777658 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:35.777622    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae8f4de_e64a_49b2_9ee2_c6e87aa06365.slice/crio-aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57 WatchSource:0}: Error finding container aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57: Status 404 returned error can't find the container with id aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57
Oct 30 22:26:35.779996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:35.779971    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"]
Oct 30 22:26:36.387715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:36.387677    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf" event={"ID":"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365","Type":"ContainerStarted","Data":"82cc80ecf6ab4f3c53a18a26ab2c26d848290964945a25b2afec063d52d3871f"}
Oct 30 22:26:36.388113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:36.387732    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf" event={"ID":"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365","Type":"ContainerStarted","Data":"aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57"}
Oct 30 22:26:36.509485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:36.509431    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf" podStartSLOduration=1.509414156 podStartE2EDuration="1.509414156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:36.50608084 +0000 UTC m=+2919.679405263" watchObservedRunningTime="2026-10-30 22:26:36.509414156 +0000 UTC m=+2919.682738628"
Oct 30 22:26:37.395292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:37.395255    4882 generic.go:356] "Generic (PLEG): container finished" podID="0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" containerID="82cc80ecf6ab4f3c53a18a26ab2c26d848290964945a25b2afec063d52d3871f" exitCode=0
Oct 30 22:26:37.395724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:37.395342    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf" event={"ID":"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365","Type":"ContainerDied","Data":"82cc80ecf6ab4f3c53a18a26ab2c26d848290964945a25b2afec063d52d3871f"}
Oct 30 22:26:38.540284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.540256    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:38.655563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.654869    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\") pod \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") "
Oct 30 22:26:38.655563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.655000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\") pod \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") "
Oct 30 22:26:38.655563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.655047    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\" (UniqueName: \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\") pod \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\" (UID: \"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365\") "
Oct 30 22:26:38.655563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.655457    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca" pod "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" (UID: "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:38.657669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.657636    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x" pod "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" (UID: "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365"). InnerVolumeSpecName "kube-api-access-54h7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:38.658292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.658261    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca" pod "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" (UID: "0ae8f4de-e64a-49b2-9ee2-c6e87aa06365"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:38.757505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.757469    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.757505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.757503    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54h7x\" (UniqueName: \"kubernetes.io/projected/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-kube-api-access-54h7x\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:38.757745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:38.757518    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:39.410412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:39.410373    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"
Oct 30 22:26:39.410412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:39.410396    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf" event={"ID":"0ae8f4de-e64a-49b2-9ee2-c6e87aa06365","Type":"ContainerDied","Data":"aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57"}
Oct 30 22:26:39.410674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:39.410451    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aab65ec967046e6129523a4f3b8185f07b018498d37b917a9e9da6bfc80e8c57"
Oct 30 22:26:42.375941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.375901    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"]
Oct 30 22:26:42.376451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.376430    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" containerName="revoke-keys"
Oct 30 22:26:42.376500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.376452    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" containerName="revoke-keys"
Oct 30 22:26:42.376560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.376550    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" containerName="revoke-keys"
Oct 30 22:26:42.380040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.380020    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.437606 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.437558    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"]
Oct 30 22:26:42.483773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.483740    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:42.483927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.483829    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:42.499041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.499005    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.499202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.499061    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.499202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.499148    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mclxm\" (UniqueName: \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.600752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.600709    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.600752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.600757    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mclxm\" (UniqueName: \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.601001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.600867    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.601389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.601360    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.603288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.603268    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.662491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.662416    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mclxm\" (UniqueName: \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\") pod \"maas-api-revoke-sub-e0b18d84-qs7k9\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:42.693851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:42.693812    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:43.066873 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:43.066821    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod479ca62c_0315_4dbb_a6ae_27a4cea765fa.slice/crio-e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f WatchSource:0}: Error finding container e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f: Status 404 returned error can't find the container with id e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f
Oct 30 22:26:43.071078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:43.071048    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"]
Oct 30 22:26:43.436009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:43.435974    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9" event={"ID":"479ca62c-0315-4dbb-a6ae-27a4cea765fa","Type":"ContainerStarted","Data":"701ebb59eae2dac4377ec1eb86109c76708d7c462ce16dd68e1358a691603192"}
Oct 30 22:26:43.436470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:43.436030    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9" event={"ID":"479ca62c-0315-4dbb-a6ae-27a4cea765fa","Type":"ContainerStarted","Data":"e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f"}
Oct 30 22:26:43.553077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:43.553020    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9" podStartSLOduration=1.553005229 podStartE2EDuration="1.553005229s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:43.552975134 +0000 UTC m=+2926.726299561" watchObservedRunningTime="2026-10-30 22:26:43.553005229 +0000 UTC m=+2926.726329655"
Oct 30 22:26:44.110536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:44.110499    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:26:44.443575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:44.443537    4882 generic.go:356] "Generic (PLEG): container finished" podID="479ca62c-0315-4dbb-a6ae-27a4cea765fa" containerID="701ebb59eae2dac4377ec1eb86109c76708d7c462ce16dd68e1358a691603192" exitCode=0
Oct 30 22:26:44.444063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:44.443628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9" event={"ID":"479ca62c-0315-4dbb-a6ae-27a4cea765fa","Type":"ContainerDied","Data":"701ebb59eae2dac4377ec1eb86109c76708d7c462ce16dd68e1358a691603192"}
Oct 30 22:26:45.612714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.612689    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:45.634516 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.634482    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\" (UniqueName: \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\") pod \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") "
Oct 30 22:26:45.634701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.634684    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\") pod \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") "
Oct 30 22:26:45.634775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.634758    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\") pod \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\" (UID: \"479ca62c-0315-4dbb-a6ae-27a4cea765fa\") "
Oct 30 22:26:45.635107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.635076    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca" pod "479ca62c-0315-4dbb-a6ae-27a4cea765fa" (UID: "479ca62c-0315-4dbb-a6ae-27a4cea765fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:45.636770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.636739    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca" pod "479ca62c-0315-4dbb-a6ae-27a4cea765fa" (UID: "479ca62c-0315-4dbb-a6ae-27a4cea765fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:45.636770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.636766    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm" pod "479ca62c-0315-4dbb-a6ae-27a4cea765fa" (UID: "479ca62c-0315-4dbb-a6ae-27a4cea765fa"). InnerVolumeSpecName "kube-api-access-mclxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:45.736634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.736536    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.736634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.736570    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/479ca62c-0315-4dbb-a6ae-27a4cea765fa-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.736634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.736581    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mclxm\" (UniqueName: \"kubernetes.io/projected/479ca62c-0315-4dbb-a6ae-27a4cea765fa-kube-api-access-mclxm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:45.882311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.882276    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"]
Oct 30 22:26:45.882822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.882807    4882 cpu_manager.go:414] "containerMap: removing container" podUID="479ca62c-0315-4dbb-a6ae-27a4cea765fa" containerName="revoke-keys"
Oct 30 22:26:45.882880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.882823    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="479ca62c-0315-4dbb-a6ae-27a4cea765fa" containerName="revoke-keys"
Oct 30 22:26:45.882937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.882928    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="479ca62c-0315-4dbb-a6ae-27a4cea765fa" containerName="revoke-keys"
Oct 30 22:26:45.886444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.886419    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:45.938654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.938619    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kwkp\" (UniqueName: \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:45.938810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.938765    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:45.938810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.938799    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:45.941210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:45.941177    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"]
Oct 30 22:26:46.039944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.039850    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.039944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.039894    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.040213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.040051    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kwkp\" (UniqueName: \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.040761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.040737    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.042443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.042417    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.103103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.103062    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kwkp\" (UniqueName: \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\") pod \"maas-api-revoke-sub-421b1b1d-6j58v\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.200786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.200747    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:46.457931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.457905    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"
Oct 30 22:26:46.458144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.457898    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9" event={"ID":"479ca62c-0315-4dbb-a6ae-27a4cea765fa","Type":"ContainerDied","Data":"e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f"}
Oct 30 22:26:46.458144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.458083    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6ab0da42e0b4b7d8bb19e1af2da78cc8669cfeb1a5d34a7acaf56c3a3ac018f"
Oct 30 22:26:46.600422 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:46.600386    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10319863_96b0_42d7_b95e_c2cff963dc4c.slice/crio-204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc WatchSource:0}: Error finding container 204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc: Status 404 returned error can't find the container with id 204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc
Oct 30 22:26:46.602678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:46.602655    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"]
Oct 30 22:26:47.014326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.014260    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" podUID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" containerName="main" containerID="cri-o://3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6" gracePeriod=45
Oct 30 22:26:47.277835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.277807    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:26:47.354518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354484    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354524    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354560    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354621    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354798    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354837    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.354910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354874    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\" (UniqueName: \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\") pod \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\" (UID: \"5bf34c12-6fe9-42c0-8f30-b669af3d94c8\") "
Oct 30 22:26:47.354910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354872    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.355031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.354898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.355302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.355271    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.355302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.355293    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.355496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.355306    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.356878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.356856    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:26:47.356978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.356881    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "kube-api-access-2z8g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:47.356978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.356941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs" pod "5bf34c12-6fe9-42c0-8f30-b669af3d94c8" (UID: "5bf34c12-6fe9-42c0-8f30-b669af3d94c8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:47.456750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.456661    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.456750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.456701    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2z8g6\" (UniqueName: \"kubernetes.io/projected/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-kube-api-access-2z8g6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.456750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.456719    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5bf34c12-6fe9-42c0-8f30-b669af3d94c8-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:47.466508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.466478    4882 generic.go:356] "Generic (PLEG): container finished" podID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" containerID="3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6" exitCode=0
Oct 30 22:26:47.466714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.466538    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"
Oct 30 22:26:47.466714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.466562    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" event={"ID":"5bf34c12-6fe9-42c0-8f30-b669af3d94c8","Type":"ContainerDied","Data":"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"}
Oct 30 22:26:47.466714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.466642    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m" event={"ID":"5bf34c12-6fe9-42c0-8f30-b669af3d94c8","Type":"ContainerDied","Data":"b9ae086cc783ada69f4f3b765c99b57fa6981225a0517f6d42ddb701a7356885"}
Oct 30 22:26:47.466714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.466662    4882 scope.go:122] "RemoveContainer" containerID="3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"
Oct 30 22:26:47.469083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.469051    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v" event={"ID":"10319863-96b0-42d7-b95e-c2cff963dc4c","Type":"ContainerStarted","Data":"0079089cbd1053437e244b28f1a7172593c2490739a43d2c73910335ac53ea3e"}
Oct 30 22:26:47.469253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.469113    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v" event={"ID":"10319863-96b0-42d7-b95e-c2cff963dc4c","Type":"ContainerStarted","Data":"204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc"}
Oct 30 22:26:47.479658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.479638    4882 scope.go:122] "RemoveContainer" containerID="3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"
Oct 30 22:26:47.479976 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:26:47.479956    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6\": container with ID starting with 3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6 not found: ID does not exist" containerID="3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"
Oct 30 22:26:47.480048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.479986    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6"} err="failed to get container status \"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6\": rpc error: code = NotFound desc = could not find container \"3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6\": container with ID starting with 3480c28d3b12b0ba785a14d390195b21995a9eb1717da75c294419c6e9b4dae6 not found: ID does not exist"
Oct 30 22:26:47.637292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.637254    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:26:47.641756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.641728    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-63a282/ipp-route-63a282-kserve-5f5f5d99cc-vx68m"]
Oct 30 22:26:47.758624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:47.758503    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v" podStartSLOduration=2.758485323 podStartE2EDuration="2.758485323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:47.756988231 +0000 UTC m=+2930.930312659" watchObservedRunningTime="2026-10-30 22:26:47.758485323 +0000 UTC m=+2930.931809751"
Oct 30 22:26:48.478455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:48.478418    4882 generic.go:356] "Generic (PLEG): container finished" podID="10319863-96b0-42d7-b95e-c2cff963dc4c" containerID="0079089cbd1053437e244b28f1a7172593c2490739a43d2c73910335ac53ea3e" exitCode=0
Oct 30 22:26:48.478913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:48.478509    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v" event={"ID":"10319863-96b0-42d7-b95e-c2cff963dc4c","Type":"ContainerDied","Data":"0079089cbd1053437e244b28f1a7172593c2490739a43d2c73910335ac53ea3e"}
Oct 30 22:26:48.943499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:48.943463    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" path="/var/lib/kubelet/pods/5bf34c12-6fe9-42c0-8f30-b669af3d94c8/volumes"
Oct 30 22:26:49.208435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.208351    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:26:49.208909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.208894    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" containerName="main"
Oct 30 22:26:49.208953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.208910    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" containerName="main"
Oct 30 22:26:49.209046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.209036    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bf34c12-6fe9-42c0-8f30-b669af3d94c8" containerName="main"
Oct 30 22:26:49.212503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.212480    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277170    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277279    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277340    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf26f\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277451    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277508    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277544    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277628    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277712    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.277807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.277790    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.323744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.323701    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:26:49.348677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.348647    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-4d14e0-openshift-default-dockercfg-mqz9f\""
Oct 30 22:26:49.378865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.378831    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.378879    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.378947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379009    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf26f\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379037    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379144    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379170    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379225    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379631    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379696    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379756    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379795    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.379989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.379933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.381543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.381520    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.381543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.381542    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.442740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.442704    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.442915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.442800    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf26f\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.530676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.530628    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:49.634949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.634917    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:49.683209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.683168    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\") pod \"10319863-96b0-42d7-b95e-c2cff963dc4c\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") "
Oct 30 22:26:49.683420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.683258    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\" (UniqueName: \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\") pod \"10319863-96b0-42d7-b95e-c2cff963dc4c\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") "
Oct 30 22:26:49.683420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.683301    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\") pod \"10319863-96b0-42d7-b95e-c2cff963dc4c\" (UID: \"10319863-96b0-42d7-b95e-c2cff963dc4c\") "
Oct 30 22:26:49.683707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.683671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca" pod "10319863-96b0-42d7-b95e-c2cff963dc4c" (UID: "10319863-96b0-42d7-b95e-c2cff963dc4c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:49.685448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.685425    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca" pod "10319863-96b0-42d7-b95e-c2cff963dc4c" (UID: "10319863-96b0-42d7-b95e-c2cff963dc4c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:49.685539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.685489    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp" pod "10319863-96b0-42d7-b95e-c2cff963dc4c" (UID: "10319863-96b0-42d7-b95e-c2cff963dc4c"). InnerVolumeSpecName "kube-api-access-9kwkp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:49.785120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.785030    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9kwkp\" (UniqueName: \"kubernetes.io/projected/10319863-96b0-42d7-b95e-c2cff963dc4c-kube-api-access-9kwkp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:49.785120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.785067    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:49.785120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.785076    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10319863-96b0-42d7-b95e-c2cff963dc4c-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:49.972581 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:49.972516    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod496e5151_612d_4e8f_bc2d_e6bd068743d7.slice/crio-0eea4cf6dac9d81f53f0ecfb52e1a83b369a99311125c0b8721563e56967c498 WatchSource:0}: Error finding container 0eea4cf6dac9d81f53f0ecfb52e1a83b369a99311125c0b8721563e56967c498: Status 404 returned error can't find the container with id 0eea4cf6dac9d81f53f0ecfb52e1a83b369a99311125c0b8721563e56967c498
Oct 30 22:26:49.975572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.975530    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:26:49.975688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.975637    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:26:49.975731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.975682    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:26:49.975893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:49.975873    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:26:50.498819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.498773    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v" event={"ID":"10319863-96b0-42d7-b95e-c2cff963dc4c","Type":"ContainerDied","Data":"204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc"}
Oct 30 22:26:50.499006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.498843    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="204e8955cecf4567a4a18eeecaa3b783dad32115bf442cc48eebf9e0a6b68bfc"
Oct 30 22:26:50.499006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.498926    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"
Oct 30 22:26:50.501065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.501032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" event={"ID":"496e5151-612d-4e8f-bc2d-e6bd068743d7","Type":"ContainerStarted","Data":"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"}
Oct 30 22:26:50.501220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.501082    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" event={"ID":"496e5151-612d-4e8f-bc2d-e6bd068743d7","Type":"ContainerStarted","Data":"0eea4cf6dac9d81f53f0ecfb52e1a83b369a99311125c0b8721563e56967c498"}
Oct 30 22:26:50.548611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.543723    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:50.648944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:50.648805    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podStartSLOduration=1.648781875 podStartE2EDuration="1.648781875s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:26:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:26:50.623871541 +0000 UTC m=+2933.797195971" watchObservedRunningTime="2026-10-30 22:26:50.648781875 +0000 UTC m=+2933.822106304"
Oct 30 22:26:50.810228 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:26:50.810186    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:26:51.549773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:51.549728    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:51.550242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:51.549797    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:52.546612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:52.540039    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:52.546612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:52.540105    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:53.538626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:53.538073    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:53.538626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:53.538138    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:54.549821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:54.549777    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:54.550316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:54.549847    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:55.544685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.536170    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:55.544685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.536239    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:55.854315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.854275    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"]
Oct 30 22:26:55.855204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.855182    4882 cpu_manager.go:414] "containerMap: removing container" podUID="10319863-96b0-42d7-b95e-c2cff963dc4c" containerName="revoke-keys"
Oct 30 22:26:55.855249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.855207    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10319863-96b0-42d7-b95e-c2cff963dc4c" containerName="revoke-keys"
Oct 30 22:26:55.855359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.855346    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="10319863-96b0-42d7-b95e-c2cff963dc4c" containerName="revoke-keys"
Oct 30 22:26:55.858722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.858705    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:55.909289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.909237    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"]
Oct 30 22:26:55.955424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.955387    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:26:55.955424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.955416    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:26:55.984399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.984360    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2s2z2\" (UniqueName: \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:55.984612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.984435    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:55.984612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:55.984495    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.085424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.085377    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2s2z2\" (UniqueName: \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.085828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.085795    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.086000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.085863    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.086680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.086652    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.088578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.088556    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.147801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.147701    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2s2z2\" (UniqueName: \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\") pod \"maas-api-revoke-sub-8af63b7f-csxq6\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.172032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.171993    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:56.531138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.531056    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused" start-of-body=
Oct 30 22:26:56.531138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.531106    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.234:15021/healthz/ready\": dial tcp 10.132.0.234:15021: connect: connection refused"
Oct 30 22:26:56.553658 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:26:56.553619    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5402867_a37f_49a3_8a7f_03a8e5ed4824.slice/crio-a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c WatchSource:0}: Error finding container a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c: Status 404 returned error can't find the container with id a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c
Oct 30 22:26:56.555846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.555821    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"]
Oct 30 22:26:56.573697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:56.573659    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6" event={"ID":"f5402867-a37f-49a3-8a7f-03a8e5ed4824","Type":"ContainerStarted","Data":"a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c"}
Oct 30 22:26:57.537225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:57.537197    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:57.537650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:57.537493    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:57.538434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:57.538413    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:26:57.582451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:57.582411    4882 generic.go:356] "Generic (PLEG): container finished" podID="f5402867-a37f-49a3-8a7f-03a8e5ed4824" containerID="69bede1820be38a0def9ee36066a0d933ae406ff5fd4adede18a1e64054ad23e" exitCode=0
Oct 30 22:26:57.582639 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:57.582498    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6" event={"ID":"f5402867-a37f-49a3-8a7f-03a8e5ed4824","Type":"ContainerDied","Data":"69bede1820be38a0def9ee36066a0d933ae406ff5fd4adede18a1e64054ad23e"}
Oct 30 22:26:58.743380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.743352    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:58.816661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.816582    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\") pod \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") "
Oct 30 22:26:58.816932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.816718    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\" (UniqueName: \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\") pod \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") "
Oct 30 22:26:58.816932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.816782    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\") pod \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\" (UID: \"f5402867-a37f-49a3-8a7f-03a8e5ed4824\") "
Oct 30 22:26:58.817619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.817558    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca" pod "f5402867-a37f-49a3-8a7f-03a8e5ed4824" (UID: "f5402867-a37f-49a3-8a7f-03a8e5ed4824"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:26:58.819298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.819265    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2" pod "f5402867-a37f-49a3-8a7f-03a8e5ed4824" (UID: "f5402867-a37f-49a3-8a7f-03a8e5ed4824"). InnerVolumeSpecName "kube-api-access-2s2z2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:26:58.819410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.819303    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca" pod "f5402867-a37f-49a3-8a7f-03a8e5ed4824" (UID: "f5402867-a37f-49a3-8a7f-03a8e5ed4824"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:26:58.918561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.918466    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:58.918561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.918501    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f5402867-a37f-49a3-8a7f-03a8e5ed4824-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:58.918561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:58.918511    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2s2z2\" (UniqueName: \"kubernetes.io/projected/f5402867-a37f-49a3-8a7f-03a8e5ed4824-kube-api-access-2s2z2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:26:59.597602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:59.597568    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"
Oct 30 22:26:59.597811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:59.597623    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6" event={"ID":"f5402867-a37f-49a3-8a7f-03a8e5ed4824","Type":"ContainerDied","Data":"a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c"}
Oct 30 22:26:59.597811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:26:59.597715    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a26fb13aaa1db131f5ffec963c8a9a510cc811c4fbc5d869254b120e28e9850c"
Oct 30 22:27:04.830228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.830191    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"]
Oct 30 22:27:04.830786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.830771    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f5402867-a37f-49a3-8a7f-03a8e5ed4824" containerName="revoke-keys"
Oct 30 22:27:04.830838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.830787    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f5402867-a37f-49a3-8a7f-03a8e5ed4824" containerName="revoke-keys"
Oct 30 22:27:04.830889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.830879    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f5402867-a37f-49a3-8a7f-03a8e5ed4824" containerName="revoke-keys"
Oct 30 22:27:04.835743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.835727    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:04.888511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.888477    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"]
Oct 30 22:27:04.934455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.934424    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:04.934455 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.934426    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:27:04.987748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.987716    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:04.987944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.987763    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:04.987989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:04.987936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxfpx\" (UniqueName: \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.088841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.088809    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.089050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.088867    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.089050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.088994    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxfpx\" (UniqueName: \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.089602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.089560    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.091496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.091472    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.150834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.150802    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxfpx\" (UniqueName: \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\") pod \"maas-api-revoke-sub-7116d5f4-q6pdf\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.449091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.449008    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:05.824140 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:05.824097    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0633118b_ec01_48d0_bc10_d3ba30db86d8.slice/crio-65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999 WatchSource:0}: Error finding container 65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999: Status 404 returned error can't find the container with id 65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999
Oct 30 22:27:05.827163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:05.827039    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"]
Oct 30 22:27:06.646790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:06.646710    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf" event={"ID":"0633118b-ec01-48d0-bc10-d3ba30db86d8","Type":"ContainerStarted","Data":"70458bfa268a4b734e6aed1105f7b189844948aaea6db48e293c3c6763f90f00"}
Oct 30 22:27:06.647288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:06.646831    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf" event={"ID":"0633118b-ec01-48d0-bc10-d3ba30db86d8","Type":"ContainerStarted","Data":"65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999"}
Oct 30 22:27:06.773396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:06.773332    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf" podStartSLOduration=2.773317311 podStartE2EDuration="2.773317311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:27:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:27:06.769914331 +0000 UTC m=+2949.943238965" watchObservedRunningTime="2026-10-30 22:27:06.773317311 +0000 UTC m=+2949.946641738"
Oct 30 22:27:07.654359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:07.654325    4882 generic.go:356] "Generic (PLEG): container finished" podID="0633118b-ec01-48d0-bc10-d3ba30db86d8" containerID="70458bfa268a4b734e6aed1105f7b189844948aaea6db48e293c3c6763f90f00" exitCode=0
Oct 30 22:27:07.654804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:07.654413    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf" event={"ID":"0633118b-ec01-48d0-bc10-d3ba30db86d8","Type":"ContainerDied","Data":"70458bfa268a4b734e6aed1105f7b189844948aaea6db48e293c3c6763f90f00"}
Oct 30 22:27:08.819990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.819961    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:08.831323 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.831292    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\") pod \"0633118b-ec01-48d0-bc10-d3ba30db86d8\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") "
Oct 30 22:27:08.831501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.831413    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\" (UniqueName: \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\") pod \"0633118b-ec01-48d0-bc10-d3ba30db86d8\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") "
Oct 30 22:27:08.831501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.831457    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\") pod \"0633118b-ec01-48d0-bc10-d3ba30db86d8\" (UID: \"0633118b-ec01-48d0-bc10-d3ba30db86d8\") "
Oct 30 22:27:08.831781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.831740    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca" pod "0633118b-ec01-48d0-bc10-d3ba30db86d8" (UID: "0633118b-ec01-48d0-bc10-d3ba30db86d8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:08.831912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.831860    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:08.833668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.833634    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca" pod "0633118b-ec01-48d0-bc10-d3ba30db86d8" (UID: "0633118b-ec01-48d0-bc10-d3ba30db86d8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:08.833782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.833734    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx" pod "0633118b-ec01-48d0-bc10-d3ba30db86d8" (UID: "0633118b-ec01-48d0-bc10-d3ba30db86d8"). InnerVolumeSpecName "kube-api-access-cxfpx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:08.935687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.935560    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxfpx\" (UniqueName: \"kubernetes.io/projected/0633118b-ec01-48d0-bc10-d3ba30db86d8-kube-api-access-cxfpx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:08.935687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:08.935635    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0633118b-ec01-48d0-bc10-d3ba30db86d8-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:09.669282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:09.669250    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"
Oct 30 22:27:09.669456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:09.669285    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf" event={"ID":"0633118b-ec01-48d0-bc10-d3ba30db86d8","Type":"ContainerDied","Data":"65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999"}
Oct 30 22:27:09.669456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:09.669343    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65f5a8db4009a34764fcff00eec93c484a4a4ce7544239e4a8709e5b94d43999"
Oct 30 22:27:17.809134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.809093    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"]
Oct 30 22:27:17.809926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.809907    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0633118b-ec01-48d0-bc10-d3ba30db86d8" containerName="revoke-keys"
Oct 30 22:27:17.810013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.809927    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0633118b-ec01-48d0-bc10-d3ba30db86d8" containerName="revoke-keys"
Oct 30 22:27:17.810077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.810063    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0633118b-ec01-48d0-bc10-d3ba30db86d8" containerName="revoke-keys"
Oct 30 22:27:17.814058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.814036    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:17.866225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.866190    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"]
Oct 30 22:27:17.867025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.866998    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:17.912990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.912944    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:27:17.933620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.933566    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7khbs\" (UniqueName: \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:17.933772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.933756    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:17.933838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:17.933802    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.034572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.034538    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.034821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.034626    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.034821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.034712    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7khbs\" (UniqueName: \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.035344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.035321    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.037006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.036985    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.097223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.097184    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7khbs\" (UniqueName: \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\") pod \"maas-api-revoke-sub-10a9200d-lqzb7\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.127795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.127758    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:18.499792 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:18.499753    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod648b24a4_8d62_48af_aa49_477f145f03c2.slice/crio-fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f WatchSource:0}: Error finding container fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f: Status 404 returned error can't find the container with id fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f
Oct 30 22:27:18.502484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.502460    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"]
Oct 30 22:27:18.729302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:18.729266    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7" event={"ID":"648b24a4-8d62-48af-aa49-477f145f03c2","Type":"ContainerStarted","Data":"fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f"}
Oct 30 22:27:19.739756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:19.739660    4882 generic.go:356] "Generic (PLEG): container finished" podID="648b24a4-8d62-48af-aa49-477f145f03c2" containerID="2c2478522bb13719e56fbb43148114993c39e0a4fb68654b9a02ef53fc202067" exitCode=0
Oct 30 22:27:19.742144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:19.739747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7" event={"ID":"648b24a4-8d62-48af-aa49-477f145f03c2","Type":"ContainerDied","Data":"2c2478522bb13719e56fbb43148114993c39e0a4fb68654b9a02ef53fc202067"}
Oct 30 22:27:20.812252 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:27:20.812139    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:27:20.914915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.914879    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:20.966257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.966228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\") pod \"648b24a4-8d62-48af-aa49-477f145f03c2\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") "
Oct 30 22:27:20.966435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.966292    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\" (UniqueName: \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\") pod \"648b24a4-8d62-48af-aa49-477f145f03c2\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") "
Oct 30 22:27:20.966489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.966442    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\") pod \"648b24a4-8d62-48af-aa49-477f145f03c2\" (UID: \"648b24a4-8d62-48af-aa49-477f145f03c2\") "
Oct 30 22:27:20.966697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.966671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca" pod "648b24a4-8d62-48af-aa49-477f145f03c2" (UID: "648b24a4-8d62-48af-aa49-477f145f03c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:20.968439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.968412    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca" pod "648b24a4-8d62-48af-aa49-477f145f03c2" (UID: "648b24a4-8d62-48af-aa49-477f145f03c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:20.968570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:20.968546    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs" pod "648b24a4-8d62-48af-aa49-477f145f03c2" (UID: "648b24a4-8d62-48af-aa49-477f145f03c2"). InnerVolumeSpecName "kube-api-access-7khbs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:21.068450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.068347    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.068450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.068386    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/648b24a4-8d62-48af-aa49-477f145f03c2-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.068450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.068402    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7khbs\" (UniqueName: \"kubernetes.io/projected/648b24a4-8d62-48af-aa49-477f145f03c2-kube-api-access-7khbs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:21.754386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.754349    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"
Oct 30 22:27:21.754386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.754380    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7" event={"ID":"648b24a4-8d62-48af-aa49-477f145f03c2","Type":"ContainerDied","Data":"fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f"}
Oct 30 22:27:21.754645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:21.754433    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fa2d01ef53b937d4095ddd79262fb16033d7ac7cd2bac34b57855577611b131f"
Oct 30 22:27:28.881696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:28.881658    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:27:41.921857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.921773    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"]
Oct 30 22:27:41.922524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.922505    4882 cpu_manager.go:414] "containerMap: removing container" podUID="648b24a4-8d62-48af-aa49-477f145f03c2" containerName="revoke-keys"
Oct 30 22:27:41.922570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.922525    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="648b24a4-8d62-48af-aa49-477f145f03c2" containerName="revoke-keys"
Oct 30 22:27:41.922678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.922667    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="648b24a4-8d62-48af-aa49-477f145f03c2" containerName="revoke-keys"
Oct 30 22:27:41.926270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.926248    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:41.978536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.978490    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"]
Oct 30 22:27:41.979289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:41.979262    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:42.019170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.019124    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.019355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.019249    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svtnt\" (UniqueName: \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.019355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.019289    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.024410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.024387    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:27:42.063607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.062171    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"]
Oct 30 22:27:42.069048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.069013    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.119991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.119955    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"]
Oct 30 22:27:42.120505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.120472    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.120834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.120806    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.121052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.121032    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq2hp\" (UniqueName: \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.121483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.121409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svtnt\" (UniqueName: \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.121483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.121470    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.121714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.121581    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.122459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.122437    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.123739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.123713    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.184875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.184811    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svtnt\" (UniqueName: \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\") pod \"maas-api-revoke-sub-0765b5c1-59kqd\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.222556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.222521    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.222749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.222576    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq2hp\" (UniqueName: \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.222749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.222669    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.223187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.223159    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.224998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.224973    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.239939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.239912    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:42.285732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.285699    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq2hp\" (UniqueName: \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\") pod \"maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.382708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.382673    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:42.612816 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:42.612783    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3560054_1d06_4c2d_ae7b_ad2a29540ea0.slice/crio-a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71 WatchSource:0}: Error finding container a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71: Status 404 returned error can't find the container with id a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71
Oct 30 22:27:42.616400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.616375    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"]
Oct 30 22:27:42.691565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.691482    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 22:27:42.691845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.691714    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" containerID="cri-o://b23b4ef869dc6d8358628edff5bd04e7ab975c396185f9d3614ecff73098215b" gracePeriod=30
Oct 30 22:27:42.705399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.705368    4882 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-7df854bf67-ksw98 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:27:42.705541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.705419    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:27:42.755304 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:42.755274    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55b9c0cb_91c9_4970_990f_3ecbf1fbf1a8.slice/crio-22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690 WatchSource:0}: Error finding container 22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690: Status 404 returned error can't find the container with id 22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690
Oct 30 22:27:42.757869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.757848    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"]
Oct 30 22:27:42.893415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.893378    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6" event={"ID":"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8","Type":"ContainerStarted","Data":"22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690"}
Oct 30 22:27:42.894829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:42.894793    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd" event={"ID":"f3560054-1d06-4c2d-ae7b-ad2a29540ea0","Type":"ContainerStarted","Data":"a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71"}
Oct 30 22:27:43.903279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:43.903245    4882 generic.go:356] "Generic (PLEG): container finished" podID="55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" containerID="cb3ee9b78e4380b99a7f4c3c55dfd23f21477dce22e0d701f711fed97244d827" exitCode=0
Oct 30 22:27:43.903800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:43.903328    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6" event={"ID":"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8","Type":"ContainerDied","Data":"cb3ee9b78e4380b99a7f4c3c55dfd23f21477dce22e0d701f711fed97244d827"}
Oct 30 22:27:43.905214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:43.905186    4882 generic.go:356] "Generic (PLEG): container finished" podID="f3560054-1d06-4c2d-ae7b-ad2a29540ea0" containerID="c10be09f42d581ce7c5ca3adc4b523691d492dabd430757bbbcbbb98bf0fb3a9" exitCode=0
Oct 30 22:27:43.905334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:43.905265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd" event={"ID":"f3560054-1d06-4c2d-ae7b-ad2a29540ea0","Type":"ContainerDied","Data":"c10be09f42d581ce7c5ca3adc4b523691d492dabd430757bbbcbbb98bf0fb3a9"}
Oct 30 22:27:45.092854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.092827    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:45.096247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.096225    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:45.158565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158530    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\" (UniqueName: \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\") pod \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") "
Oct 30 22:27:45.158565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158568    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\" (UniqueName: \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\") pod \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") "
Oct 30 22:27:45.158876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158622    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\") pod \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") "
Oct 30 22:27:45.158876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158721    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\") pod \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\" (UID: \"f3560054-1d06-4c2d-ae7b-ad2a29540ea0\") "
Oct 30 22:27:45.158876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158749    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\") pod \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") "
Oct 30 22:27:45.158876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.158817    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\") pod \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\" (UID: \"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8\") "
Oct 30 22:27:45.159144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.159112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca" pod "f3560054-1d06-4c2d-ae7b-ad2a29540ea0" (UID: "f3560054-1d06-4c2d-ae7b-ad2a29540ea0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:45.159144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.159133    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca" pod "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" (UID: "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:45.159268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.159244    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.159340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.159273    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.160951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.160924    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca" pod "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" (UID: "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:45.161084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.161002    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp" pod "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" (UID: "55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8"). InnerVolumeSpecName "kube-api-access-rq2hp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:45.161304 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.161277    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca" pod "f3560054-1d06-4c2d-ae7b-ad2a29540ea0" (UID: "f3560054-1d06-4c2d-ae7b-ad2a29540ea0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:45.161375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.161285    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt" pod "f3560054-1d06-4c2d-ae7b-ad2a29540ea0" (UID: "f3560054-1d06-4c2d-ae7b-ad2a29540ea0"). InnerVolumeSpecName "kube-api-access-svtnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:45.203616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.203291    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"]
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.204945    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f3560054-1d06-4c2d-ae7b-ad2a29540ea0" containerName="revoke-keys"
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.204974    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3560054-1d06-4c2d-ae7b-ad2a29540ea0" containerName="revoke-keys"
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.204988    4882 cpu_manager.go:414] "containerMap: removing container" podUID="55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" containerName="revoke-keys"
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.204996    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" containerName="revoke-keys"
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.205286    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3560054-1d06-4c2d-ae7b-ad2a29540ea0" containerName="revoke-keys"
Oct 30 22:27:45.207803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.205311    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" containerName="revoke-keys"
Oct 30 22:27:45.210351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.210325    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260419    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260560    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2tkm\" (UniqueName: \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260652    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260767    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260784    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260800    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rq2hp\" (UniqueName: \"kubernetes.io/projected/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8-kube-api-access-rq2hp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.260913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.260814    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-svtnt\" (UniqueName: \"kubernetes.io/projected/f3560054-1d06-4c2d-ae7b-ad2a29540ea0-kube-api-access-svtnt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:45.263040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.263003    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"]
Oct 30 22:27:45.362127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.362096    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.362350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.362213    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.362350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.362317    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2tkm\" (UniqueName: \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.362998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.362974    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.364579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.364561    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.425472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.425398    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2tkm\" (UniqueName: \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\") pod \"maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.525082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.525045    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:45.896915 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:45.896879    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39ae35fb_020b_4b88_8107_4321b7fd66a1.slice/crio-159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7 WatchSource:0}: Error finding container 159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7: Status 404 returned error can't find the container with id 159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7
Oct 30 22:27:45.899873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.899847    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"]
Oct 30 22:27:45.931015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.930995    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"
Oct 30 22:27:45.931203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.931013    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6" event={"ID":"55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8","Type":"ContainerDied","Data":"22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690"}
Oct 30 22:27:45.931203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.931080    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="22aaf443988237c4f0e5f42ceaf7811be182ade49b5d4d0d013395b9c4ef4690"
Oct 30 22:27:45.932505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.932476    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l" event={"ID":"39ae35fb-020b-4b88-8107-4321b7fd66a1","Type":"ContainerStarted","Data":"159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7"}
Oct 30 22:27:45.933965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.933941    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd" event={"ID":"f3560054-1d06-4c2d-ae7b-ad2a29540ea0","Type":"ContainerDied","Data":"a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71"}
Oct 30 22:27:45.934080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.933979    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a453a771c4146887bfbb011c9adb0975e2b206842c4767b9b00c814f65728b71"
Oct 30 22:27:45.934080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:45.933946    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"
Oct 30 22:27:46.943234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:46.943204    4882 generic.go:356] "Generic (PLEG): container finished" podID="39ae35fb-020b-4b88-8107-4321b7fd66a1" containerID="ef72278ca8406220253020335010232a782cd87a7590b898c6328f3d8b467897" exitCode=0
Oct 30 22:27:46.944277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:46.944240    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l" event={"ID":"39ae35fb-020b-4b88-8107-4321b7fd66a1","Type":"ContainerDied","Data":"ef72278ca8406220253020335010232a782cd87a7590b898c6328f3d8b467897"}
Oct 30 22:27:47.953047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:47.952979    4882 generic.go:356] "Generic (PLEG): container finished" podID="a92a87fb-b018-468f-bbd6-328c15452c94" containerID="b23b4ef869dc6d8358628edff5bd04e7ab975c396185f9d3614ecff73098215b" exitCode=0
Oct 30 22:27:47.953503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:47.953472    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" event={"ID":"a92a87fb-b018-468f-bbd6-328c15452c94","Type":"ContainerDied","Data":"b23b4ef869dc6d8358628edff5bd04e7ab975c396185f9d3614ecff73098215b"}
Oct 30 22:27:47.953602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:47.953535    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98" event={"ID":"a92a87fb-b018-468f-bbd6-328c15452c94","Type":"ContainerDied","Data":"9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673"}
Oct 30 22:27:47.953602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:47.953562    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c738718b77cc40d87f0e0f7005ea825649acf2b19921b14184c0b948bd21673"
Oct 30 22:27:47.972400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:47.972370    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 22:27:48.094937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.094910    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.094965    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095044    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095074    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095148    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095198    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:48.095335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095222    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095311    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\") pod \"a92a87fb-b018-468f-bbd6-328c15452c94\" (UID: \"a92a87fb-b018-468f-bbd6-328c15452c94\") "
Oct 30 22:27:48.095620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095563    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:48.095710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095686    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:48.095775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095747    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:48.095887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095858    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:48.095952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095922    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.095952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095943    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.096063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095958    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.096063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.095972    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.097453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.097425    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "kube-api-access-nmftp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:48.097610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.097557    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:48.097715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.097696    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:27:48.097765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.097709    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy" pod "a92a87fb-b018-468f-bbd6-328c15452c94" (UID: "a92a87fb-b018-468f-bbd6-328c15452c94"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:27:48.110014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.109991    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:48.197008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.196971    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\") pod \"39ae35fb-020b-4b88-8107-4321b7fd66a1\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") "
Oct 30 22:27:48.197008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197012    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\") pod \"39ae35fb-020b-4b88-8107-4321b7fd66a1\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") "
Oct 30 22:27:48.197231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197154    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\" (UniqueName: \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\") pod \"39ae35fb-020b-4b88-8107-4321b7fd66a1\" (UID: \"39ae35fb-020b-4b88-8107-4321b7fd66a1\") "
Oct 30 22:27:48.197404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197381    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca" pod "39ae35fb-020b-4b88-8107-4321b7fd66a1" (UID: "39ae35fb-020b-4b88-8107-4321b7fd66a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:48.197452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197435    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a92a87fb-b018-468f-bbd6-328c15452c94-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.197452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197449    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a92a87fb-b018-468f-bbd6-328c15452c94-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.197522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197458    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.197522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197467    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmftp\" (UniqueName: \"kubernetes.io/projected/a92a87fb-b018-468f-bbd6-328c15452c94-kube-api-access-nmftp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.197522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.197479    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a92a87fb-b018-468f-bbd6-328c15452c94-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.199239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.199217    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm" pod "39ae35fb-020b-4b88-8107-4321b7fd66a1" (UID: "39ae35fb-020b-4b88-8107-4321b7fd66a1"). InnerVolumeSpecName "kube-api-access-t2tkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:48.199313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.199292    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca" pod "39ae35fb-020b-4b88-8107-4321b7fd66a1" (UID: "39ae35fb-020b-4b88-8107-4321b7fd66a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:48.304335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.304252    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.304335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.304291    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39ae35fb-020b-4b88-8107-4321b7fd66a1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.304335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.304304    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2tkm\" (UniqueName: \"kubernetes.io/projected/39ae35fb-020b-4b88-8107-4321b7fd66a1-kube-api-access-t2tkm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:48.961537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.961511    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"
Oct 30 22:27:48.961907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.961540    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l" event={"ID":"39ae35fb-020b-4b88-8107-4321b7fd66a1","Type":"ContainerDied","Data":"159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7"}
Oct 30 22:27:48.961907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.961604    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="159a6a1ebc0d7b6dc481b4a7d51033841acea67463a1e91b8a8a2de3492bfbe7"
Oct 30 22:27:48.961907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:48.961684    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"
Oct 30 22:27:49.184031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:49.183993    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 22:27:49.234427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:49.234396    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-7df854bf67-ksw98"]
Oct 30 22:27:50.809180 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:27:50.808203    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:27:50.943258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:50.943222    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" path="/var/lib/kubelet/pods/a92a87fb-b018-468f-bbd6-328c15452c94/volumes"
Oct 30 22:27:55.252246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252211    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"]
Oct 30 22:27:55.252773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252758    4882 cpu_manager.go:414] "containerMap: removing container" podUID="39ae35fb-020b-4b88-8107-4321b7fd66a1" containerName="revoke-keys"
Oct 30 22:27:55.252828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252775    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39ae35fb-020b-4b88-8107-4321b7fd66a1" containerName="revoke-keys"
Oct 30 22:27:55.252828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252805    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy"
Oct 30 22:27:55.252828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252811    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy"
Oct 30 22:27:55.252927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252895    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a92a87fb-b018-468f-bbd6-328c15452c94" containerName="istio-proxy"
Oct 30 22:27:55.252927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.252908    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="39ae35fb-020b-4b88-8107-4321b7fd66a1" containerName="revoke-keys"
Oct 30 22:27:55.257696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.257676    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.311778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.311741    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"]
Oct 30 22:27:55.357911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.357873    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:27:55.358092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.357934    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:27:55.381414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.381380    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.381414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.381413    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62m8\" (UniqueName: \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.381712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.381457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.482546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.482504    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.482546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.482546    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62m8\" (UniqueName: \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.482826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.482619    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.483349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.483318    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.485203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.485180    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.544346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.544268    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62m8\" (UniqueName: \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\") pod \"maas-api-revoke-sub-6d81b914-nvh5d\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.571203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.571163    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:55.947186 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:55.947153    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fe0a43c_6a9d_4fc5_9ce3_80be390d8b4f.slice/crio-b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e WatchSource:0}: Error finding container b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e: Status 404 returned error can't find the container with id b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e
Oct 30 22:27:55.950213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:55.950183    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"]
Oct 30 22:27:56.006124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:56.006085    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d" event={"ID":"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f","Type":"ContainerStarted","Data":"b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e"}
Oct 30 22:27:56.666251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:56.666213    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:27:57.017709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.017614    4882 generic.go:356] "Generic (PLEG): container finished" podID="6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" containerID="0849ff8459b95e0a3e305e4ed37e9516f59e8d968a2913fdfcdc0a7fbbac0742" exitCode=0
Oct 30 22:27:57.017709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.017669    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d" event={"ID":"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f","Type":"ContainerDied","Data":"0849ff8459b95e0a3e305e4ed37e9516f59e8d968a2913fdfcdc0a7fbbac0742"}
Oct 30 22:27:57.288106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.288028    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"]
Oct 30 22:27:57.292693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.292667    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.348095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.348061    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"]
Oct 30 22:27:57.368181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.368150    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:27:57.382381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.382356    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:27:57.409082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.409050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.409265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.409092    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.409265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.409234    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-76ggv\" (UniqueName: \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.510696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.510654    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.510891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.510704    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.510891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.510810    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-76ggv\" (UniqueName: \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.511426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.511397    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.513128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.513101    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.572745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.572675    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-76ggv\" (UniqueName: \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\") pod \"maas-api-revoke-sub-732fc0fd-6jj6w\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.607599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.607557    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:27:57.984375 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:27:57.984340    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398fc6aa_9510_461a_a45d_2898e6f42118.slice/crio-2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74 WatchSource:0}: Error finding container 2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74: Status 404 returned error can't find the container with id 2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74
Oct 30 22:27:57.987176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:57.987150    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"]
Oct 30 22:27:58.026634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.026568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w" event={"ID":"398fc6aa-9510-461a-a45d-2898e6f42118","Type":"ContainerStarted","Data":"2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74"}
Oct 30 22:27:58.156443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.156413    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:58.320528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.320487    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\" (UniqueName: \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\") pod \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") "
Oct 30 22:27:58.320528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.320535    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\") pod \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") "
Oct 30 22:27:58.320760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.320623    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\") pod \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\" (UID: \"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f\") "
Oct 30 22:27:58.320983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.320946    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca" pod "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" (UID: "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:27:58.321130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.321114    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.322987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.322960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca" pod "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" (UID: "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:27:58.322987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.322979    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8" pod "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" (UID: "6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f"). InnerVolumeSpecName "kube-api-access-n62m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:27:58.422210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.422177    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:58.422210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:58.422206    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62m8\" (UniqueName: \"kubernetes.io/projected/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f-kube-api-access-n62m8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:27:59.036253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.036222    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"
Oct 30 22:27:59.036692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.036219    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d" event={"ID":"6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f","Type":"ContainerDied","Data":"b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e"}
Oct 30 22:27:59.036692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.036342    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b4c64f1b6c7aa953a3c03705a8550a4639af26fa3bb2820bb603ede15fcfce6e"
Oct 30 22:27:59.037935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.037908    4882 generic.go:356] "Generic (PLEG): container finished" podID="398fc6aa-9510-461a-a45d-2898e6f42118" containerID="3071c6d0a62377bdee2c16da23bc27c1b3b107100a0e79e4d68e44b2a7614e3d" exitCode=0
Oct 30 22:27:59.038084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.037990    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w" event={"ID":"398fc6aa-9510-461a-a45d-2898e6f42118","Type":"ContainerDied","Data":"3071c6d0a62377bdee2c16da23bc27c1b3b107100a0e79e4d68e44b2a7614e3d"}
Oct 30 22:27:59.085442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:27:59.085409    4882 scope.go:122] "RemoveContainer" containerID="b23b4ef869dc6d8358628edff5bd04e7ab975c396185f9d3614ecff73098215b"
Oct 30 22:28:00.200125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.200093    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:28:00.343485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.343453    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\") pod \"398fc6aa-9510-461a-a45d-2898e6f42118\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") "
Oct 30 22:28:00.343750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.343499    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\") pod \"398fc6aa-9510-461a-a45d-2898e6f42118\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") "
Oct 30 22:28:00.343750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.343673    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\" (UniqueName: \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\") pod \"398fc6aa-9510-461a-a45d-2898e6f42118\" (UID: \"398fc6aa-9510-461a-a45d-2898e6f42118\") "
Oct 30 22:28:00.343969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.343932    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca" pod "398fc6aa-9510-461a-a45d-2898e6f42118" (UID: "398fc6aa-9510-461a-a45d-2898e6f42118"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:00.344147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.344129    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:00.345845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.345814    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca" pod "398fc6aa-9510-461a-a45d-2898e6f42118" (UID: "398fc6aa-9510-461a-a45d-2898e6f42118"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:00.345946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.345861    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv" pod "398fc6aa-9510-461a-a45d-2898e6f42118" (UID: "398fc6aa-9510-461a-a45d-2898e6f42118"). InnerVolumeSpecName "kube-api-access-76ggv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:00.445411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.445370    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/398fc6aa-9510-461a-a45d-2898e6f42118-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:00.445411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:00.445406    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-76ggv\" (UniqueName: \"kubernetes.io/projected/398fc6aa-9510-461a-a45d-2898e6f42118-kube-api-access-76ggv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:01.056598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:01.056566    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"
Oct 30 22:28:01.056752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:01.056573    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w" event={"ID":"398fc6aa-9510-461a-a45d-2898e6f42118","Type":"ContainerDied","Data":"2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74"}
Oct 30 22:28:01.056752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:01.056701    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c255e7ab6dea12047422c610786a02d26483fbaba00132644764c6739f15f74"
Oct 30 22:28:01.666719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:01.666675    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing" containerID="cri-o://2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92" gracePeriod=25
Oct 30 22:28:02.437484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:02.437444    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:28:02.753073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:02.752983    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:28:02.753575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:02.753208    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" podUID="23c02225-8584-47fe-8e58-f7ff75b8aa26" containerName="payload-processing" containerID="cri-o://5c3f509a5814403b5feef10016a601e68ea98d8f78e17b5672a814ea60228635" gracePeriod=30
Oct 30 22:28:02.805610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:02.805559    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:28:02.805828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:02.805780    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" podUID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" containerName="payload-pre-processing" containerID="cri-o://95f7ef9b127b305f773280f70dc2c1ee9d3df9dc1fd94a2f36ab2acdb295e574" gracePeriod=30
Oct 30 22:28:03.075232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.075189    4882 generic.go:356] "Generic (PLEG): container finished" podID="23c02225-8584-47fe-8e58-f7ff75b8aa26" containerID="5c3f509a5814403b5feef10016a601e68ea98d8f78e17b5672a814ea60228635" exitCode=0
Oct 30 22:28:03.075441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.075293    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" event={"ID":"23c02225-8584-47fe-8e58-f7ff75b8aa26","Type":"ContainerDied","Data":"5c3f509a5814403b5feef10016a601e68ea98d8f78e17b5672a814ea60228635"}
Oct 30 22:28:03.075441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.075344    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59" event={"ID":"23c02225-8584-47fe-8e58-f7ff75b8aa26","Type":"ContainerDied","Data":"80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e"}
Oct 30 22:28:03.075441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.075370    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="80a458ea1cb0b44f4325989288570283cf5cbbd6ad4f126884b3555fe62ff02e"
Oct 30 22:28:03.078908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.078862    4882 generic.go:356] "Generic (PLEG): container finished" podID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" containerID="95f7ef9b127b305f773280f70dc2c1ee9d3df9dc1fd94a2f36ab2acdb295e574" exitCode=0
Oct 30 22:28:03.079052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.078938    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" event={"ID":"f14d2645-8c0b-4c92-8cda-5d5aba187bc7","Type":"ContainerDied","Data":"95f7ef9b127b305f773280f70dc2c1ee9d3df9dc1fd94a2f36ab2acdb295e574"}
Oct 30 22:28:03.095294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.095263    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:28:03.117380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.117355    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:28:03.280099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280001    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\") pod \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") "
Oct 30 22:28:03.280099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280066    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\" (UniqueName: \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\") pod \"23c02225-8584-47fe-8e58-f7ff75b8aa26\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") "
Oct 30 22:28:03.280427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280146    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\" (UniqueName: \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\") pod \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\" (UID: \"f14d2645-8c0b-4c92-8cda-5d5aba187bc7\") "
Oct 30 22:28:03.280427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280266    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\") pod \"23c02225-8584-47fe-8e58-f7ff75b8aa26\" (UID: \"23c02225-8584-47fe-8e58-f7ff75b8aa26\") "
Oct 30 22:28:03.280427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280410    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume" pod "f14d2645-8c0b-4c92-8cda-5d5aba187bc7" (UID: "f14d2645-8c0b-4c92-8cda-5d5aba187bc7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:03.280706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280685    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume" pod "23c02225-8584-47fe-8e58-f7ff75b8aa26" (UID: "23c02225-8584-47fe-8e58-f7ff75b8aa26"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:03.280831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280814    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.280873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.280837    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23c02225-8584-47fe-8e58-f7ff75b8aa26-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.282285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.282261    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7" pod "f14d2645-8c0b-4c92-8cda-5d5aba187bc7" (UID: "f14d2645-8c0b-4c92-8cda-5d5aba187bc7"). InnerVolumeSpecName "kube-api-access-mm4x7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.282357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.282345    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r" pod "23c02225-8584-47fe-8e58-f7ff75b8aa26" (UID: "23c02225-8584-47fe-8e58-f7ff75b8aa26"). InnerVolumeSpecName "kube-api-access-cnm6r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:03.382431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.382387    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnm6r\" (UniqueName: \"kubernetes.io/projected/23c02225-8584-47fe-8e58-f7ff75b8aa26-kube-api-access-cnm6r\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:03.382431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:03.382430    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm4x7\" (UniqueName: \"kubernetes.io/projected/f14d2645-8c0b-4c92-8cda-5d5aba187bc7-kube-api-access-mm4x7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:04.087521 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.087491    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"
Oct 30 22:28:04.088013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.087534    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz" event={"ID":"f14d2645-8c0b-4c92-8cda-5d5aba187bc7","Type":"ContainerDied","Data":"359785f94ac753711e4df53ca5e7631829ae1a7e13ecee2c1894fd21be9319ed"}
Oct 30 22:28:04.088013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.087613    4882 scope.go:122] "RemoveContainer" containerID="95f7ef9b127b305f773280f70dc2c1ee9d3df9dc1fd94a2f36ab2acdb295e574"
Oct 30 22:28:04.088013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.087662    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"
Oct 30 22:28:04.366997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.366912    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:28:04.380565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.380534    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-d2f1cf-bd68dfd75-njtbz"]
Oct 30 22:28:04.493927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.493879    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:28:04.509626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.509594    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-d2f1cf-69dd76d744-nfc59"]
Oct 30 22:28:04.755861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.755781    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"]
Oct 30 22:28:04.766314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.766283    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-d2f1cf-jrzw6"]
Oct 30 22:28:04.942534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.942485    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="23c02225-8584-47fe-8e58-f7ff75b8aa26" path="/var/lib/kubelet/pods/23c02225-8584-47fe-8e58-f7ff75b8aa26/volumes"
Oct 30 22:28:04.943062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.943040    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8" path="/var/lib/kubelet/pods/55b9c0cb-91c9-4970-990f-3ecbf1fbf1a8/volumes"
Oct 30 22:28:04.943490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:04.943470    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" path="/var/lib/kubelet/pods/f14d2645-8c0b-4c92-8cda-5d5aba187bc7/volumes"
Oct 30 22:28:07.038331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038298    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"]
Oct 30 22:28:07.038809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038794    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" containerName="revoke-keys"
Oct 30 22:28:07.038872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038811    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" containerName="revoke-keys"
Oct 30 22:28:07.038872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038843    4882 cpu_manager.go:414] "containerMap: removing container" podUID="23c02225-8584-47fe-8e58-f7ff75b8aa26" containerName="payload-processing"
Oct 30 22:28:07.038872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038849    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23c02225-8584-47fe-8e58-f7ff75b8aa26" containerName="payload-processing"
Oct 30 22:28:07.038872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038857    4882 cpu_manager.go:414] "containerMap: removing container" podUID="398fc6aa-9510-461a-a45d-2898e6f42118" containerName="revoke-keys"
Oct 30 22:28:07.038872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038862    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398fc6aa-9510-461a-a45d-2898e6f42118" containerName="revoke-keys"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038876    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" containerName="payload-pre-processing"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038882    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" containerName="payload-pre-processing"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038980    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="23c02225-8584-47fe-8e58-f7ff75b8aa26" containerName="payload-processing"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038987    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="398fc6aa-9510-461a-a45d-2898e6f42118" containerName="revoke-keys"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.038996    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f14d2645-8c0b-4c92-8cda-5d5aba187bc7" containerName="payload-pre-processing"
Oct 30 22:28:07.039038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.039004    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" containerName="revoke-keys"
Oct 30 22:28:07.043615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.043577    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.097019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.096977    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"]
Oct 30 22:28:07.120420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.120384    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.120661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.120431    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2m2\" (UniqueName: \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.120661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.120533    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.129574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.129542    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:07.129741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.129577    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:28:07.222197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.222161    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.222417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.222207    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2m2\" (UniqueName: \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.222417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.222248    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.222924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.222900    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.224708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.224689    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.284448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.284413    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2m2\" (UniqueName: \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\") pod \"maas-api-revoke-sub-5977c5c3-nzt57\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.357743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.357714    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:07.735075 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:07.735037    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf28780d4_c968_4a31_a718_48fdc7e5dd3b.slice/crio-b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c WatchSource:0}: Error finding container b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c: Status 404 returned error can't find the container with id b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c
Oct 30 22:28:07.738195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:07.738168    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"]
Oct 30 22:28:08.116432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:08.116385    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57" event={"ID":"f28780d4-c968-4a31-a718-48fdc7e5dd3b","Type":"ContainerStarted","Data":"e3fbe0a0a1522661dae12ab652a2dfa18f317764c7098b80e314c37fe6e8c7dd"}
Oct 30 22:28:08.116878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:08.116448    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57" event={"ID":"f28780d4-c968-4a31-a718-48fdc7e5dd3b","Type":"ContainerStarted","Data":"b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c"}
Oct 30 22:28:08.240047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:08.239995    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57" podStartSLOduration=2.239979212 podStartE2EDuration="2.239979212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:08.236026666 +0000 UTC m=+3011.409351092" watchObservedRunningTime="2026-10-30 22:28:08.239979212 +0000 UTC m=+3011.413303639"
Oct 30 22:28:09.125202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:09.125164    4882 generic.go:356] "Generic (PLEG): container finished" podID="f28780d4-c968-4a31-a718-48fdc7e5dd3b" containerID="e3fbe0a0a1522661dae12ab652a2dfa18f317764c7098b80e314c37fe6e8c7dd" exitCode=0
Oct 30 22:28:09.125637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:09.125250    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57" event={"ID":"f28780d4-c968-4a31-a718-48fdc7e5dd3b","Type":"ContainerDied","Data":"e3fbe0a0a1522661dae12ab652a2dfa18f317764c7098b80e314c37fe6e8c7dd"}
Oct 30 22:28:10.276964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.276939    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:10.361266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.361234    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\" (UniqueName: \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\") pod \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") "
Oct 30 22:28:10.361457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.361320    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\") pod \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") "
Oct 30 22:28:10.361457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.361428    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\") pod \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\" (UID: \"f28780d4-c968-4a31-a718-48fdc7e5dd3b\") "
Oct 30 22:28:10.361802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.361778    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca" pod "f28780d4-c968-4a31-a718-48fdc7e5dd3b" (UID: "f28780d4-c968-4a31-a718-48fdc7e5dd3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:10.363308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.363282    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2" pod "f28780d4-c968-4a31-a718-48fdc7e5dd3b" (UID: "f28780d4-c968-4a31-a718-48fdc7e5dd3b"). InnerVolumeSpecName "kube-api-access-wm2m2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:10.363466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.363445    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca" pod "f28780d4-c968-4a31-a718-48fdc7e5dd3b" (UID: "f28780d4-c968-4a31-a718-48fdc7e5dd3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:10.462505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.462415    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm2m2\" (UniqueName: \"kubernetes.io/projected/f28780d4-c968-4a31-a718-48fdc7e5dd3b-kube-api-access-wm2m2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:10.462505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.462453    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:10.462505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.462463    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f28780d4-c968-4a31-a718-48fdc7e5dd3b-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:10.627312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:10.627276    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:11.141135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:11.141103    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"
Oct 30 22:28:11.141289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:11.141132    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57" event={"ID":"f28780d4-c968-4a31-a718-48fdc7e5dd3b","Type":"ContainerDied","Data":"b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c"}
Oct 30 22:28:11.141289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:11.141188    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b109b26bc499a8da0b387369725c42f433e7fbb3a1b735e46657e80a44d0153c"
Oct 30 22:28:18.644813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.644776    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"]
Oct 30 22:28:18.645542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.645520    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f28780d4-c968-4a31-a718-48fdc7e5dd3b" containerName="revoke-keys"
Oct 30 22:28:18.645707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.645545    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f28780d4-c968-4a31-a718-48fdc7e5dd3b" containerName="revoke-keys"
Oct 30 22:28:18.645776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.645729    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f28780d4-c968-4a31-a718-48fdc7e5dd3b" containerName="revoke-keys"
Oct 30 22:28:18.649999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.649979    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.703410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.703371    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"]
Oct 30 22:28:18.749549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.749510    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:18.749549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.749544    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:28:18.756211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.756178    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.756369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.756293    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bs8nc\" (UniqueName: \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.756487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.756468    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.813902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.813862    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"]
Oct 30 22:28:18.822168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.822131    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:18.857289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.857253    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.857469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.857336    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.857469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.857393    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bs8nc\" (UniqueName: \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.858098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.858069    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.859668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.859645    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.873345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.873310    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"]
Oct 30 22:28:18.919780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.919704    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bs8nc\" (UniqueName: \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\") pod \"maas-api-revoke-sub-511ea286-dvwgb\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:18.958136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.958101    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd8wm\" (UniqueName: \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:18.958321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.958147    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:18.958374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.958328    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:18.963917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:18.963897    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:19.059343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.059298    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.059538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.059391    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd8wm\" (UniqueName: \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.059538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.059485    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.060077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.060042    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.061977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.061951    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.121723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.121687    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd8wm\" (UniqueName: \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\") pod \"maas-api-revoke-sub-a6946e2f-4mpdh\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.138256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.138225    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:19.345885 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:19.345842    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod149523c7_a00a_4a3f_90bc_9a90f9cb0695.slice/crio-28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245 WatchSource:0}: Error finding container 28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245: Status 404 returned error can't find the container with id 28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245
Oct 30 22:28:19.347836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.347802    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"]
Oct 30 22:28:19.514843 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:19.514804    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod218e8468_e91e_42d5_82ab_7af4eaae0afc.slice/crio-c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231 WatchSource:0}: Error finding container c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231: Status 404 returned error can't find the container with id c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231
Oct 30 22:28:19.519016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:19.518992    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"]
Oct 30 22:28:20.217059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.217019    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh" event={"ID":"218e8468-e91e-42d5-82ab-7af4eaae0afc","Type":"ContainerStarted","Data":"cb1ff8bb7ec3ac972e38be387309678de71b36fbf34ffd9e80b70995dbc9457f"}
Oct 30 22:28:20.217509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.217091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh" event={"ID":"218e8468-e91e-42d5-82ab-7af4eaae0afc","Type":"ContainerStarted","Data":"c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231"}
Oct 30 22:28:20.218738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.218712    4882 generic.go:356] "Generic (PLEG): container finished" podID="149523c7-a00a-4a3f-90bc-9a90f9cb0695" containerID="03965f536f2e29a315f82aa2cbda1f540c8f3ddf17fef383291c412b5645cba0" exitCode=0
Oct 30 22:28:20.218861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.218793    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb" event={"ID":"149523c7-a00a-4a3f-90bc-9a90f9cb0695","Type":"ContainerDied","Data":"03965f536f2e29a315f82aa2cbda1f540c8f3ddf17fef383291c412b5645cba0"}
Oct 30 22:28:20.218861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.218850    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb" event={"ID":"149523c7-a00a-4a3f-90bc-9a90f9cb0695","Type":"ContainerStarted","Data":"28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245"}
Oct 30 22:28:20.336211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.336154    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh" podStartSLOduration=2.336137902 podStartE2EDuration="2.336137902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:20.333135014 +0000 UTC m=+3023.506459441" watchObservedRunningTime="2026-10-30 22:28:20.336137902 +0000 UTC m=+3023.509462328"
Oct 30 22:28:20.627999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:20.627965    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:28:20.809984 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:28:20.809898    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:28:21.228617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.228573    4882 generic.go:356] "Generic (PLEG): container finished" podID="218e8468-e91e-42d5-82ab-7af4eaae0afc" containerID="cb1ff8bb7ec3ac972e38be387309678de71b36fbf34ffd9e80b70995dbc9457f" exitCode=0
Oct 30 22:28:21.228617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.228614    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh" event={"ID":"218e8468-e91e-42d5-82ab-7af4eaae0afc","Type":"ContainerDied","Data":"cb1ff8bb7ec3ac972e38be387309678de71b36fbf34ffd9e80b70995dbc9457f"}
Oct 30 22:28:21.382211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.382177    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:21.487419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.487326    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\" (UniqueName: \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\") pod \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") "
Oct 30 22:28:21.487625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.487427    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\") pod \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") "
Oct 30 22:28:21.487625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.487470    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\") pod \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\" (UID: \"149523c7-a00a-4a3f-90bc-9a90f9cb0695\") "
Oct 30 22:28:21.487867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.487834    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca" pod "149523c7-a00a-4a3f-90bc-9a90f9cb0695" (UID: "149523c7-a00a-4a3f-90bc-9a90f9cb0695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:21.489682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.489657    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca" pod "149523c7-a00a-4a3f-90bc-9a90f9cb0695" (UID: "149523c7-a00a-4a3f-90bc-9a90f9cb0695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:21.489770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.489684    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc" pod "149523c7-a00a-4a3f-90bc-9a90f9cb0695" (UID: "149523c7-a00a-4a3f-90bc-9a90f9cb0695"). InnerVolumeSpecName "kube-api-access-bs8nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:21.588413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.588379    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bs8nc\" (UniqueName: \"kubernetes.io/projected/149523c7-a00a-4a3f-90bc-9a90f9cb0695-kube-api-access-bs8nc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:21.588413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.588410    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:21.588633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.588422    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/149523c7-a00a-4a3f-90bc-9a90f9cb0695-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:21.937547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.937519    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:28:21.992771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.992741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\" (UniqueName: \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\") pod \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") "
Oct 30 22:28:21.992961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.992893    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\") pod \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\" (UID: \"6fbfd9b6-462b-4427-9d1f-a76f337ef956\") "
Oct 30 22:28:21.993341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.993315    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume" pod "6fbfd9b6-462b-4427-9d1f-a76f337ef956" (UID: "6fbfd9b6-462b-4427-9d1f-a76f337ef956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:21.994889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:21.994868    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf" pod "6fbfd9b6-462b-4427-9d1f-a76f337ef956" (UID: "6fbfd9b6-462b-4427-9d1f-a76f337ef956"). InnerVolumeSpecName "kube-api-access-jkcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:22.094269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.094235    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fbfd9b6-462b-4427-9d1f-a76f337ef956-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:22.094269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.094266    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jkcvf\" (UniqueName: \"kubernetes.io/projected/6fbfd9b6-462b-4427-9d1f-a76f337ef956-kube-api-access-jkcvf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:22.109078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109043    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"]
Oct 30 22:28:22.109531 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109518    4882 cpu_manager.go:414] "containerMap: removing container" podUID="149523c7-a00a-4a3f-90bc-9a90f9cb0695" containerName="revoke-keys"
Oct 30 22:28:22.109609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109532    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="149523c7-a00a-4a3f-90bc-9a90f9cb0695" containerName="revoke-keys"
Oct 30 22:28:22.109609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109563    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing"
Oct 30 22:28:22.109609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109568    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing"
Oct 30 22:28:22.109724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109671    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerName="payload-processing"
Oct 30 22:28:22.109724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.109685    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="149523c7-a00a-4a3f-90bc-9a90f9cb0695" containerName="revoke-keys"
Oct 30 22:28:22.113123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.113106    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.123057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.123022    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"]
Oct 30 22:28:22.194896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.194864    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.195056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.194914    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.195115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.195053    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vvvg\" (UniqueName: \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.237061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.237019    4882 generic.go:356] "Generic (PLEG): container finished" podID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" containerID="2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92" exitCode=0
Oct 30 22:28:22.237510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.237071    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"
Oct 30 22:28:22.237510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.237102    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" event={"ID":"6fbfd9b6-462b-4427-9d1f-a76f337ef956","Type":"ContainerDied","Data":"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"}
Oct 30 22:28:22.237510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.237158    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn" event={"ID":"6fbfd9b6-462b-4427-9d1f-a76f337ef956","Type":"ContainerDied","Data":"fdbadd301502744798f15fc0f51c9e939a464a0f6898dedf8b61d30890f169ce"}
Oct 30 22:28:22.237510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.237183    4882 scope.go:122] "RemoveContainer" containerID="2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"
Oct 30 22:28:22.238648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.238506    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"
Oct 30 22:28:22.238648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.238530    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb" event={"ID":"149523c7-a00a-4a3f-90bc-9a90f9cb0695","Type":"ContainerDied","Data":"28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245"}
Oct 30 22:28:22.238648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.238609    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28ffbd47a119e2612f0be3282466c9b8b5b910eb6ed7f66d0591ab8926a2b245"
Oct 30 22:28:22.249345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.249324    4882 scope.go:122] "RemoveContainer" containerID="2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"
Oct 30 22:28:22.249891 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:28:22.249720    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92\": container with ID starting with 2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92 not found: ID does not exist" containerID="2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"
Oct 30 22:28:22.249891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.249762    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92"} err="failed to get container status \"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92\": rpc error: code = NotFound desc = could not find container \"2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92\": container with ID starting with 2cb75280c293c1681235ad590051b26d96910d8d05f71217ef812eedcc9eeb92 not found: ID does not exist"
Oct 30 22:28:22.296088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.296056    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vvvg\" (UniqueName: \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.296238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.296204    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.296307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.296263    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.298223 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.297112    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.302813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.299812    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.361094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.361067    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vvvg\" (UniqueName: \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\") pod \"maas-api-revoke-sub-a72476f6-ljsrz\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.384364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.384335    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:22.429386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.429353    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:22.459809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.459771    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:28:22.463193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.463165    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-a-f7c488-8f5b95d7-5zffn"]
Oct 30 22:28:22.498258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.498224    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\" (UniqueName: \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\") pod \"218e8468-e91e-42d5-82ab-7af4eaae0afc\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") "
Oct 30 22:28:22.498431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.498302    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\") pod \"218e8468-e91e-42d5-82ab-7af4eaae0afc\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") "
Oct 30 22:28:22.498431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.498338    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\") pod \"218e8468-e91e-42d5-82ab-7af4eaae0afc\" (UID: \"218e8468-e91e-42d5-82ab-7af4eaae0afc\") "
Oct 30 22:28:22.499002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.498969    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca" pod "218e8468-e91e-42d5-82ab-7af4eaae0afc" (UID: "218e8468-e91e-42d5-82ab-7af4eaae0afc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:22.500937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.500909    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca" pod "218e8468-e91e-42d5-82ab-7af4eaae0afc" (UID: "218e8468-e91e-42d5-82ab-7af4eaae0afc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:22.504374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.504340    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm" pod "218e8468-e91e-42d5-82ab-7af4eaae0afc" (UID: "218e8468-e91e-42d5-82ab-7af4eaae0afc"). InnerVolumeSpecName "kube-api-access-sd8wm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:22.599459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.599423    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd8wm\" (UniqueName: \"kubernetes.io/projected/218e8468-e91e-42d5-82ab-7af4eaae0afc-kube-api-access-sd8wm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:22.599459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.599455    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:22.599671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.599467    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/218e8468-e91e-42d5-82ab-7af4eaae0afc-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:22.807084 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:22.807051    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28c50176_9c1f_450f_af80_7e4c60015e37.slice/crio-c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2 WatchSource:0}: Error finding container c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2: Status 404 returned error can't find the container with id c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2
Oct 30 22:28:22.811431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.811404    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"]
Oct 30 22:28:22.873571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.873528    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"]
Oct 30 22:28:22.874101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.874080    4882 cpu_manager.go:414] "containerMap: removing container" podUID="218e8468-e91e-42d5-82ab-7af4eaae0afc" containerName="revoke-keys"
Oct 30 22:28:22.874101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.874098    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="218e8468-e91e-42d5-82ab-7af4eaae0afc" containerName="revoke-keys"
Oct 30 22:28:22.874283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.874205    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="218e8468-e91e-42d5-82ab-7af4eaae0afc" containerName="revoke-keys"
Oct 30 22:28:22.878901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.878875    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:22.946627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.946558    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fbfd9b6-462b-4427-9d1f-a76f337ef956" path="/var/lib/kubelet/pods/6fbfd9b6-462b-4427-9d1f-a76f337ef956/volumes"
Oct 30 22:28:22.947291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:22.947272    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"]
Oct 30 22:28:23.004270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.004223    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.004453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.004384    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.004602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.004573    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92jf4\" (UniqueName: \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.105699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.105657    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92jf4\" (UniqueName: \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.105924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.105738    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.105924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.105809    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.106491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.106465    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.108129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.108111    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.167684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.167643    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-92jf4\" (UniqueName: \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\") pod \"maas-api-revoke-sub-e7a0be76-8ntc9\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.194511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.194478    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:23.247336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.247058    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"
Oct 30 22:28:23.247336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.247087    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh" event={"ID":"218e8468-e91e-42d5-82ab-7af4eaae0afc","Type":"ContainerDied","Data":"c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231"}
Oct 30 22:28:23.247336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.247150    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c43bee91e2907a4d8c724f763154fdbd8ac79412a8b80e55f7c80ba4c88e9231"
Oct 30 22:28:23.253902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.252138    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz" event={"ID":"28c50176-9c1f-450f-af80-7e4c60015e37","Type":"ContainerStarted","Data":"b8f0983f9948b2b007ec7e0af9bac86718936305485214a44684a2b3e99e68a4"}
Oct 30 22:28:23.253902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.252188    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz" event={"ID":"28c50176-9c1f-450f-af80-7e4c60015e37","Type":"ContainerStarted","Data":"c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2"}
Oct 30 22:28:23.432183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.432076    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz" podStartSLOduration=1.432060304 podStartE2EDuration="1.432060304s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:23.429182304 +0000 UTC m=+3026.602506730" watchObservedRunningTime="2026-10-30 22:28:23.432060304 +0000 UTC m=+3026.605384731"
Oct 30 22:28:23.569967 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:23.569917    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95ef4854_825e_4247_a406_ae6ecc12d109.slice/crio-5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43 WatchSource:0}: Error finding container 5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43: Status 404 returned error can't find the container with id 5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43
Oct 30 22:28:23.574246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:23.574217    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"]
Oct 30 22:28:24.261163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:24.261129    4882 generic.go:356] "Generic (PLEG): container finished" podID="28c50176-9c1f-450f-af80-7e4c60015e37" containerID="b8f0983f9948b2b007ec7e0af9bac86718936305485214a44684a2b3e99e68a4" exitCode=0
Oct 30 22:28:24.261640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:24.261211    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz" event={"ID":"28c50176-9c1f-450f-af80-7e4c60015e37","Type":"ContainerDied","Data":"b8f0983f9948b2b007ec7e0af9bac86718936305485214a44684a2b3e99e68a4"}
Oct 30 22:28:24.262844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:24.262819    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9" event={"ID":"95ef4854-825e-4247-a406-ae6ecc12d109","Type":"ContainerStarted","Data":"c5bf7641294cfd41a745063df9ebdb65481d869c7f25ebaac6bc3548e77db592"}
Oct 30 22:28:24.262968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:24.262857    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9" event={"ID":"95ef4854-825e-4247-a406-ae6ecc12d109","Type":"ContainerStarted","Data":"5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43"}
Oct 30 22:28:24.499913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:24.499864    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9" podStartSLOduration=2.499813493 podStartE2EDuration="2.499813493s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:24.49794077 +0000 UTC m=+3027.671265197" watchObservedRunningTime="2026-10-30 22:28:24.499813493 +0000 UTC m=+3027.673137967"
Oct 30 22:28:25.206930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.206888    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:28:25.210992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.210971    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.264844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.264812    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-3c4802-openshift-default-dockercfg-6gcqv\""
Oct 30 22:28:25.271069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.271039    4882 generic.go:356] "Generic (PLEG): container finished" podID="95ef4854-825e-4247-a406-ae6ecc12d109" containerID="c5bf7641294cfd41a745063df9ebdb65481d869c7f25ebaac6bc3548e77db592" exitCode=0
Oct 30 22:28:25.271249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.271129    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9" event={"ID":"95ef4854-825e-4247-a406-ae6ecc12d109","Type":"ContainerDied","Data":"c5bf7641294cfd41a745063df9ebdb65481d869c7f25ebaac6bc3548e77db592"}
Oct 30 22:28:25.320902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.320870    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:28:25.326817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326555    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.326817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326681    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.326817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326744    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58j6\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326885    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326928    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326954    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.326991    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.327047    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.327106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.327075    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427673 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427645    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427683    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427704    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427725    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427762    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.427801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427790    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427867    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427934    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.427976    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58j6\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.428451    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.428489    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.428522    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.428567    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.428929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.428907    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.430677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.430655    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.430677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.430670    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.450448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.450416    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:25.489883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.489794    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.490062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.489874    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58j6\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.525045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.525015    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:25.529314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.529290    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\") pod \"28c50176-9c1f-450f-af80-7e4c60015e37\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") "
Oct 30 22:28:25.529432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.529418    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\" (UniqueName: \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\") pod \"28c50176-9c1f-450f-af80-7e4c60015e37\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") "
Oct 30 22:28:25.529475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.529454    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\") pod \"28c50176-9c1f-450f-af80-7e4c60015e37\" (UID: \"28c50176-9c1f-450f-af80-7e4c60015e37\") "
Oct 30 22:28:25.529608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.529574    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca" pod "28c50176-9c1f-450f-af80-7e4c60015e37" (UID: "28c50176-9c1f-450f-af80-7e4c60015e37"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:25.529826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.529806    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.531514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.531489    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca" pod "28c50176-9c1f-450f-af80-7e4c60015e37" (UID: "28c50176-9c1f-450f-af80-7e4c60015e37"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:25.531638 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.531600    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg" pod "28c50176-9c1f-450f-af80-7e4c60015e37" (UID: "28c50176-9c1f-450f-af80-7e4c60015e37"). InnerVolumeSpecName "kube-api-access-8vvvg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:25.631357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.631317    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vvvg\" (UniqueName: \"kubernetes.io/projected/28c50176-9c1f-450f-af80-7e4c60015e37-kube-api-access-8vvvg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.631357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.631360    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/28c50176-9c1f-450f-af80-7e4c60015e37-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:25.957721 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:25.957682    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372d3d7c_84d3_4a4d_9088_e1c1061ad657.slice/crio-18da2b969466e41a4cc7b097ac26fe70a3456a38cc149e79dd344d7e561eb295 WatchSource:0}: Error finding container 18da2b969466e41a4cc7b097ac26fe70a3456a38cc149e79dd344d7e561eb295: Status 404 returned error can't find the container with id 18da2b969466e41a4cc7b097ac26fe70a3456a38cc149e79dd344d7e561eb295
Oct 30 22:28:25.960457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.960421    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:25.960548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.960495    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:25.960548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.960525    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:25.961550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:25.961523    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:28:26.279562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.279465    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" event={"ID":"372d3d7c-84d3-4a4d-9088-e1c1061ad657","Type":"ContainerStarted","Data":"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"}
Oct 30 22:28:26.279562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.279530    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" event={"ID":"372d3d7c-84d3-4a4d-9088-e1c1061ad657","Type":"ContainerStarted","Data":"18da2b969466e41a4cc7b097ac26fe70a3456a38cc149e79dd344d7e561eb295"}
Oct 30 22:28:26.281085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.281065    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"
Oct 30 22:28:26.281228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.281108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz" event={"ID":"28c50176-9c1f-450f-af80-7e4c60015e37","Type":"ContainerDied","Data":"c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2"}
Oct 30 22:28:26.281228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.281145    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f5624626ab9b6307c29373e2f5c4dfe8f492802f8a00b42a006d68927781c2"
Oct 30 22:28:26.404472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.404407    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podStartSLOduration=1.404387231 podStartE2EDuration="1.404387231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:26.402964863 +0000 UTC m=+3029.576289291" watchObservedRunningTime="2026-10-30 22:28:26.404387231 +0000 UTC m=+3029.577711659"
Oct 30 22:28:26.447579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.447550    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:26.525326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.525283    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:26.542862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.542771    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\") pod \"95ef4854-825e-4247-a406-ae6ecc12d109\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") "
Oct 30 22:28:26.543022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.542912    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\" (UniqueName: \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\") pod \"95ef4854-825e-4247-a406-ae6ecc12d109\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") "
Oct 30 22:28:26.543084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.543054    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\") pod \"95ef4854-825e-4247-a406-ae6ecc12d109\" (UID: \"95ef4854-825e-4247-a406-ae6ecc12d109\") "
Oct 30 22:28:26.543411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.543383    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca" pod "95ef4854-825e-4247-a406-ae6ecc12d109" (UID: "95ef4854-825e-4247-a406-ae6ecc12d109"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:26.544982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.544960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4" pod "95ef4854-825e-4247-a406-ae6ecc12d109" (UID: "95ef4854-825e-4247-a406-ae6ecc12d109"). InnerVolumeSpecName "kube-api-access-92jf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:26.545092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.544985    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca" pod "95ef4854-825e-4247-a406-ae6ecc12d109" (UID: "95ef4854-825e-4247-a406-ae6ecc12d109"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:26.645281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.645243    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-92jf4\" (UniqueName: \"kubernetes.io/projected/95ef4854-825e-4247-a406-ae6ecc12d109-kube-api-access-92jf4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:26.645281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.645278    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:26.645281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:26.645292    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/95ef4854-825e-4247-a406-ae6ecc12d109-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:27.290060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:27.289978    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"
Oct 30 22:28:27.290060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:27.290000    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9" event={"ID":"95ef4854-825e-4247-a406-ae6ecc12d109","Type":"ContainerDied","Data":"5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43"}
Oct 30 22:28:27.290060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:27.290050    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5bb61a3cdedd5653972296167414e850f7ea755e469d1e467af19b4a42369f43"
Oct 30 22:28:27.551524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:27.551422    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:27.551524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:27.551491    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:28.546613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:28.541304    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:28.546613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:28.541366    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:29.541612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:29.529023    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:29.541612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:29.529088    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:30.544616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:30.540831    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:30.544616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:30.540899    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:31.540613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:31.531062    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:31.540613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:31.531171    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:32.526565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.526520    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused" start-of-body=
Oct 30 22:28:32.526998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.526620    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.248:15021/healthz/ready\": dial tcp 10.132.0.248:15021: connect: connection refused"
Oct 30 22:28:32.748768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.748732    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"]
Oct 30 22:28:32.749249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749233    4882 cpu_manager.go:414] "containerMap: removing container" podUID="28c50176-9c1f-450f-af80-7e4c60015e37" containerName="revoke-keys"
Oct 30 22:28:32.749249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749249    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c50176-9c1f-450f-af80-7e4c60015e37" containerName="revoke-keys"
Oct 30 22:28:32.749397 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749267    4882 cpu_manager.go:414] "containerMap: removing container" podUID="95ef4854-825e-4247-a406-ae6ecc12d109" containerName="revoke-keys"
Oct 30 22:28:32.749397 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749273    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="95ef4854-825e-4247-a406-ae6ecc12d109" containerName="revoke-keys"
Oct 30 22:28:32.749505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749402    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c50176-9c1f-450f-af80-7e4c60015e37" containerName="revoke-keys"
Oct 30 22:28:32.749505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.749416    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="95ef4854-825e-4247-a406-ae6ecc12d109" containerName="revoke-keys"
Oct 30 22:28:32.753251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.753229    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.816232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.816140    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.816232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.816188    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.816232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.816221    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nfxd4\" (UniqueName: \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.818938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.818906    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"]
Oct 30 22:28:32.852649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.852609    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:32.852829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.852663    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:28:32.917454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.917404    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.917454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.917458    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.917719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.917493    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nfxd4\" (UniqueName: \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.918581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.918555    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.921509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.921484    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:32.979915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:32.979881    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nfxd4\" (UniqueName: \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\") pod \"maas-api-revoke-sub-44f13885-sdtxh\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:33.067975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:33.067889    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:33.448764 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:33.448718    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd636af08_7e3d_41bd_b0da_3d8160149812.slice/crio-e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f WatchSource:0}: Error finding container e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f: Status 404 returned error can't find the container with id e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f
Oct 30 22:28:33.450433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:33.450404    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"]
Oct 30 22:28:33.812881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:33.812837    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:33.814316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:33.814164    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:33.815564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:33.815540    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:28:34.400919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:34.400884    4882 generic.go:356] "Generic (PLEG): container finished" podID="d636af08-7e3d-41bd-b0da-3d8160149812" containerID="648de49670db1016de7a61e81dc700dfe9720977e00184ba3635a575fec45a57" exitCode=0
Oct 30 22:28:34.401111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:34.400976    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh" event={"ID":"d636af08-7e3d-41bd-b0da-3d8160149812","Type":"ContainerDied","Data":"648de49670db1016de7a61e81dc700dfe9720977e00184ba3635a575fec45a57"}
Oct 30 22:28:34.401111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:34.401066    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh" event={"ID":"d636af08-7e3d-41bd-b0da-3d8160149812","Type":"ContainerStarted","Data":"e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f"}
Oct 30 22:28:35.557126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.557096    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:35.658383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.658346    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\") pod \"d636af08-7e3d-41bd-b0da-3d8160149812\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") "
Oct 30 22:28:35.658547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.658416    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\" (UniqueName: \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\") pod \"d636af08-7e3d-41bd-b0da-3d8160149812\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") "
Oct 30 22:28:35.658724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.658545    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\") pod \"d636af08-7e3d-41bd-b0da-3d8160149812\" (UID: \"d636af08-7e3d-41bd-b0da-3d8160149812\") "
Oct 30 22:28:35.658890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.658823    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca" pod "d636af08-7e3d-41bd-b0da-3d8160149812" (UID: "d636af08-7e3d-41bd-b0da-3d8160149812"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:35.660702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.660676    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca" pod "d636af08-7e3d-41bd-b0da-3d8160149812" (UID: "d636af08-7e3d-41bd-b0da-3d8160149812"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:35.660790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.660678    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4" pod "d636af08-7e3d-41bd-b0da-3d8160149812" (UID: "d636af08-7e3d-41bd-b0da-3d8160149812"). InnerVolumeSpecName "kube-api-access-nfxd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:35.760091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.760059    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nfxd4\" (UniqueName: \"kubernetes.io/projected/d636af08-7e3d-41bd-b0da-3d8160149812-kube-api-access-nfxd4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.760091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.760087    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:35.760091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:35.760098    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d636af08-7e3d-41bd-b0da-3d8160149812-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:36.416680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.416643    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh" event={"ID":"d636af08-7e3d-41bd-b0da-3d8160149812","Type":"ContainerDied","Data":"e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f"}
Oct 30 22:28:36.416842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.416700    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e56bfae3c57743f825e11c44dedd229579ce86188405dbaf5cffad37178c786f"
Oct 30 22:28:36.416842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.416660    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"
Oct 30 22:28:36.944308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.944278    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"]
Oct 30 22:28:36.944804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.944787    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d636af08-7e3d-41bd-b0da-3d8160149812" containerName="revoke-keys"
Oct 30 22:28:36.944847 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.944806    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d636af08-7e3d-41bd-b0da-3d8160149812" containerName="revoke-keys"
Oct 30 22:28:36.944901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.944893    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d636af08-7e3d-41bd-b0da-3d8160149812" containerName="revoke-keys"
Oct 30 22:28:36.948471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:36.948447    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.001631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.001597    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"]
Oct 30 22:28:37.002903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.001932    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:37.002903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.002056    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:28:37.074628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.074563    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.074820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.074715    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.074958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.074939    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlgfp\" (UniqueName: \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.175669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.175626    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlgfp\" (UniqueName: \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.175669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.175676    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.175897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.175714    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.176375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.176355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.178211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.178188    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.238052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.237979    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlgfp\" (UniqueName: \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\") pod \"maas-api-revoke-sub-a17cce00-9cvkt\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.262859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.262824    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:37.636037 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:37.636005    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcab449cb_c32f_4452_8ede_00eaf2f35a34.slice/crio-e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948 WatchSource:0}: Error finding container e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948: Status 404 returned error can't find the container with id e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948
Oct 30 22:28:37.639278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:37.639256    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"]
Oct 30 22:28:38.432961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:38.432921    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt" event={"ID":"cab449cb-c32f-4452-8ede-00eaf2f35a34","Type":"ContainerStarted","Data":"ef4c63df06eabffd04859833dd4ea5d7c2e530d08b88df6003731d881281403f"}
Oct 30 22:28:38.433401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:38.433016    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt" event={"ID":"cab449cb-c32f-4452-8ede-00eaf2f35a34","Type":"ContainerStarted","Data":"e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948"}
Oct 30 22:28:38.550728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:38.550666    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt" podStartSLOduration=2.550651706 podStartE2EDuration="2.550651706s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:38.549020542 +0000 UTC m=+3041.722344967" watchObservedRunningTime="2026-10-30 22:28:38.550651706 +0000 UTC m=+3041.723976132"
Oct 30 22:28:39.442301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:39.442266    4882 generic.go:356] "Generic (PLEG): container finished" podID="cab449cb-c32f-4452-8ede-00eaf2f35a34" containerID="ef4c63df06eabffd04859833dd4ea5d7c2e530d08b88df6003731d881281403f" exitCode=0
Oct 30 22:28:39.442736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:39.442346    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt" event={"ID":"cab449cb-c32f-4452-8ede-00eaf2f35a34","Type":"ContainerDied","Data":"ef4c63df06eabffd04859833dd4ea5d7c2e530d08b88df6003731d881281403f"}
Oct 30 22:28:40.589621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.589578    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:40.716284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.716195    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\" (UniqueName: \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\") pod \"cab449cb-c32f-4452-8ede-00eaf2f35a34\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") "
Oct 30 22:28:40.716284 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.716263    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\") pod \"cab449cb-c32f-4452-8ede-00eaf2f35a34\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") "
Oct 30 22:28:40.716514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.716474    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\") pod \"cab449cb-c32f-4452-8ede-00eaf2f35a34\" (UID: \"cab449cb-c32f-4452-8ede-00eaf2f35a34\") "
Oct 30 22:28:40.716919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.716884    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca" pod "cab449cb-c32f-4452-8ede-00eaf2f35a34" (UID: "cab449cb-c32f-4452-8ede-00eaf2f35a34"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:40.718321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.718287    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca" pod "cab449cb-c32f-4452-8ede-00eaf2f35a34" (UID: "cab449cb-c32f-4452-8ede-00eaf2f35a34"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:40.718792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.718771    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp" pod "cab449cb-c32f-4452-8ede-00eaf2f35a34" (UID: "cab449cb-c32f-4452-8ede-00eaf2f35a34"). InnerVolumeSpecName "kube-api-access-rlgfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:40.817515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.817483    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rlgfp\" (UniqueName: \"kubernetes.io/projected/cab449cb-c32f-4452-8ede-00eaf2f35a34-kube-api-access-rlgfp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.817515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.817515    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:40.817756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:40.817530    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cab449cb-c32f-4452-8ede-00eaf2f35a34-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:41.458252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:41.458218    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"
Oct 30 22:28:41.458445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:41.458247    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt" event={"ID":"cab449cb-c32f-4452-8ede-00eaf2f35a34","Type":"ContainerDied","Data":"e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948"}
Oct 30 22:28:41.458445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:41.458301    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2e49441986c1ac21e8009184227000c0ab6687b612a3479a254723702d76948"
Oct 30 22:28:43.447600 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.447553    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"]
Oct 30 22:28:43.448141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.448121    4882 cpu_manager.go:414] "containerMap: removing container" podUID="cab449cb-c32f-4452-8ede-00eaf2f35a34" containerName="revoke-keys"
Oct 30 22:28:43.448226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.448144    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cab449cb-c32f-4452-8ede-00eaf2f35a34" containerName="revoke-keys"
Oct 30 22:28:43.448293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.448276    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="cab449cb-c32f-4452-8ede-00eaf2f35a34" containerName="revoke-keys"
Oct 30 22:28:43.453551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.453513    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.506654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.506617    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"]
Oct 30 22:28:43.508795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.508768    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:28:43.547385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.547350    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rt4qm\" (UniqueName: \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.547569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.547408    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.547569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.547530    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.549134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.549112    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:28:43.649157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.649115    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rt4qm\" (UniqueName: \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.649379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.649181    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.649379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.649231    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.649889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.649861    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.651650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.651629    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.711637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.711541    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rt4qm\" (UniqueName: \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\") pod \"maas-api-revoke-sub-d260d2a5-bkdhn\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:43.771755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:43.771722    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:44.015616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.015527    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"]
Oct 30 22:28:44.020926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.020904    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.075030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.074994    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"]
Oct 30 22:28:44.155117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.155067    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k4pj\" (UniqueName: \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.155302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.155218    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.155302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.155289    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.177823 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:44.177783    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7d722b80_0dba_41eb_a237_cf79931643d1.slice/crio-7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e WatchSource:0}: Error finding container 7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e: Status 404 returned error can't find the container with id 7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e
Oct 30 22:28:44.181425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.181397    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"]
Oct 30 22:28:44.256081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.256041    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.256266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.256134    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.256266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.256233    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k4pj\" (UniqueName: \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.256813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.256791    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.258415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.258397    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.319028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.318951    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k4pj\" (UniqueName: \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\") pod \"maas-api-revoke-sub-cc261a01-clqrr\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.336825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.336795    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:44.480493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.480455    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn" event={"ID":"7d722b80-0dba-41eb-a237-cf79931643d1","Type":"ContainerStarted","Data":"7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e"}
Oct 30 22:28:44.711229 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:44.711196    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b17afd6_1430_455e_a927_05bd45497c3b.slice/crio-00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba WatchSource:0}: Error finding container 00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba: Status 404 returned error can't find the container with id 00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba
Oct 30 22:28:44.715342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:44.715314    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"]
Oct 30 22:28:45.488807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:45.488768    4882 generic.go:356] "Generic (PLEG): container finished" podID="7d722b80-0dba-41eb-a237-cf79931643d1" containerID="91e39e66fa687ecd54318ee8ad6d21355e4a2aca10f0abe4e1312973edb0881c" exitCode=0
Oct 30 22:28:45.489271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:45.488858    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn" event={"ID":"7d722b80-0dba-41eb-a237-cf79931643d1","Type":"ContainerDied","Data":"91e39e66fa687ecd54318ee8ad6d21355e4a2aca10f0abe4e1312973edb0881c"}
Oct 30 22:28:45.490711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:45.490688    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr" event={"ID":"8b17afd6-1430-455e-a927-05bd45497c3b","Type":"ContainerStarted","Data":"24613bc186f80f260d23de3e060e280e819a834962d8c788d31c7f096bb06912"}
Oct 30 22:28:45.490840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:45.490733    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr" event={"ID":"8b17afd6-1430-455e-a927-05bd45497c3b","Type":"ContainerStarted","Data":"00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba"}
Oct 30 22:28:45.722814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:45.722760    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr" podStartSLOduration=2.72274353 podStartE2EDuration="2.72274353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:45.721618349 +0000 UTC m=+3048.894942779" watchObservedRunningTime="2026-10-30 22:28:45.72274353 +0000 UTC m=+3048.896067958"
Oct 30 22:28:46.500413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.500376    4882 generic.go:356] "Generic (PLEG): container finished" podID="8b17afd6-1430-455e-a927-05bd45497c3b" containerID="24613bc186f80f260d23de3e060e280e819a834962d8c788d31c7f096bb06912" exitCode=0
Oct 30 22:28:46.500861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.500468    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr" event={"ID":"8b17afd6-1430-455e-a927-05bd45497c3b","Type":"ContainerDied","Data":"24613bc186f80f260d23de3e060e280e819a834962d8c788d31c7f096bb06912"}
Oct 30 22:28:46.703478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.703439    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:46.887795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.887763    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\" (UniqueName: \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\") pod \"7d722b80-0dba-41eb-a237-cf79931643d1\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") "
Oct 30 22:28:46.887995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.887851    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\") pod \"7d722b80-0dba-41eb-a237-cf79931643d1\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") "
Oct 30 22:28:46.888111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.887995    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\") pod \"7d722b80-0dba-41eb-a237-cf79931643d1\" (UID: \"7d722b80-0dba-41eb-a237-cf79931643d1\") "
Oct 30 22:28:46.888414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.888388    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca" pod "7d722b80-0dba-41eb-a237-cf79931643d1" (UID: "7d722b80-0dba-41eb-a237-cf79931643d1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:46.890065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.890033    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm" pod "7d722b80-0dba-41eb-a237-cf79931643d1" (UID: "7d722b80-0dba-41eb-a237-cf79931643d1"). InnerVolumeSpecName "kube-api-access-rt4qm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:46.890253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.890223    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca" pod "7d722b80-0dba-41eb-a237-cf79931643d1" (UID: "7d722b80-0dba-41eb-a237-cf79931643d1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:46.990005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.989957    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:46.990005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.990003    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rt4qm\" (UniqueName: \"kubernetes.io/projected/7d722b80-0dba-41eb-a237-cf79931643d1-kube-api-access-rt4qm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:46.990242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:46.990022    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7d722b80-0dba-41eb-a237-cf79931643d1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.511018 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.510970    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn" event={"ID":"7d722b80-0dba-41eb-a237-cf79931643d1","Type":"ContainerDied","Data":"7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e"}
Oct 30 22:28:47.511555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.511041    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7235c3d01699eb21bb00c0e8d860818952d49c29b9aa5ef97df74fe4d0e5ec6e"
Oct 30 22:28:47.511555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.510987    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"
Oct 30 22:28:47.675481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.675452    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:47.696311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.696279    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\") pod \"8b17afd6-1430-455e-a927-05bd45497c3b\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") "
Oct 30 22:28:47.698429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.698403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca" pod "8b17afd6-1430-455e-a927-05bd45497c3b" (UID: "8b17afd6-1430-455e-a927-05bd45497c3b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:28:47.797260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.797164    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\") pod \"8b17afd6-1430-455e-a927-05bd45497c3b\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") "
Oct 30 22:28:47.797436 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.797263    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\" (UniqueName: \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\") pod \"8b17afd6-1430-455e-a927-05bd45497c3b\" (UID: \"8b17afd6-1430-455e-a927-05bd45497c3b\") "
Oct 30 22:28:47.797680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.797658    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.798471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.798120    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca" pod "8b17afd6-1430-455e-a927-05bd45497c3b" (UID: "8b17afd6-1430-455e-a927-05bd45497c3b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:28:47.800281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.800250    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj" pod "8b17afd6-1430-455e-a927-05bd45497c3b" (UID: "8b17afd6-1430-455e-a927-05bd45497c3b"). InnerVolumeSpecName "kube-api-access-2k4pj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:28:47.898787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.898756    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b17afd6-1430-455e-a927-05bd45497c3b-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:47.898787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:47.898787    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k4pj\" (UniqueName: \"kubernetes.io/projected/8b17afd6-1430-455e-a927-05bd45497c3b-kube-api-access-2k4pj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:28:48.519067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:48.519034    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"
Oct 30 22:28:48.519500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:48.519074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr" event={"ID":"8b17afd6-1430-455e-a927-05bd45497c3b","Type":"ContainerDied","Data":"00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba"}
Oct 30 22:28:48.519500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:48.519124    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00eb0cfc369561674eb255c39581ebc25c021db2d20ffcee2c861dcc1daa0bba"
Oct 30 22:28:50.164329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.164280    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:28:50.173429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173391    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8b17afd6-1430-455e-a927-05bd45497c3b" containerName="revoke-keys"
Oct 30 22:28:50.173637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173426    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b17afd6-1430-455e-a927-05bd45497c3b" containerName="revoke-keys"
Oct 30 22:28:50.173637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173524    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7d722b80-0dba-41eb-a237-cf79931643d1" containerName="revoke-keys"
Oct 30 22:28:50.173637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173533    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7d722b80-0dba-41eb-a237-cf79931643d1" containerName="revoke-keys"
Oct 30 22:28:50.173810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173794    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b17afd6-1430-455e-a927-05bd45497c3b" containerName="revoke-keys"
Oct 30 22:28:50.173847 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.173816    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7d722b80-0dba-41eb-a237-cf79931643d1" containerName="revoke-keys"
Oct 30 22:28:50.178781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.178753    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.211278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.211243    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:28:50.215607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.215564    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.221300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221259    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.221466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221312    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.221466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221339    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.221466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221391    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.221466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221422    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9sh4h\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.221466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221460    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221497    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221549    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221579    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221621    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nf6qr\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221675    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221773    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221822    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221887    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221919    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.227935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.221953    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.290883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.290846    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:28:50.296202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.296163    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:28:50.322925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.322880    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.322925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.322922    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.322946    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.322984    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323002    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323017    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323053    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323084    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9sh4h\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323124    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323159    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323208    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323235    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323257    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nf6qr\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323308    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323331    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323390    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323393    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323408    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323432    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323487    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.323773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323414    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323854    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323892    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.323960    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.324271    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.324281    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.324396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.324358    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.326568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.326539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.326888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.326866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.327024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.327002    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.327125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.327087    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.332297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.332274    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-56f3fe-openshift-default-dockercfg-48wbx\""
Oct 30 22:28:50.388358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.388313    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.388577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.388556    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.388682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.388663    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nf6qr\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\") pod \"e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") " pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.393732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.393705    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9sh4h\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\") pod \"e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") " pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.494858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.494771    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:50.534749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.534718    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:50.811165 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:28:50.811079    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:28:50.923029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.922999    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:28:50.952922 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:50.952888    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fb98270_b727_419a_a50c_ebce91f90863.slice/crio-93db18008ba9487f0017d4dcdbe35c491792751df0648ac8e8d60990eb768d6b WatchSource:0}: Error finding container 93db18008ba9487f0017d4dcdbe35c491792751df0648ac8e8d60990eb768d6b: Status 404 returned error can't find the container with id 93db18008ba9487f0017d4dcdbe35c491792751df0648ac8e8d60990eb768d6b
Oct 30 22:28:50.955277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.955243    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:50.955382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.955312    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:50.955382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.955339    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:50.981247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.981218    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:28:50.989607 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:50.989554    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf95518bf_37c2_446d_a964_aaa56b988b0b.slice/crio-3213eb2ec58146f288c8cba905ec3a5de39758fea162ace95558f66aea4900fd WatchSource:0}: Error finding container 3213eb2ec58146f288c8cba905ec3a5de39758fea162ace95558f66aea4900fd: Status 404 returned error can't find the container with id 3213eb2ec58146f288c8cba905ec3a5de39758fea162ace95558f66aea4900fd
Oct 30 22:28:50.992293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.992256    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:50.992385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.992356    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:50.992433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:50.992402    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:51.559306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.559259    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" event={"ID":"8fb98270-b727-419a-a50c-ebce91f90863","Type":"ContainerStarted","Data":"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"}
Oct 30 22:28:51.559848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.559341    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" event={"ID":"8fb98270-b727-419a-a50c-ebce91f90863","Type":"ContainerStarted","Data":"93db18008ba9487f0017d4dcdbe35c491792751df0648ac8e8d60990eb768d6b"}
Oct 30 22:28:51.561671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.561598    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" event={"ID":"f95518bf-37c2-446d-a964-aaa56b988b0b","Type":"ContainerStarted","Data":"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"}
Oct 30 22:28:51.561671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.561647    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" event={"ID":"f95518bf-37c2-446d-a964-aaa56b988b0b","Type":"ContainerStarted","Data":"3213eb2ec58146f288c8cba905ec3a5de39758fea162ace95558f66aea4900fd"}
Oct 30 22:28:51.688415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.688361    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podStartSLOduration=1.688346712 podStartE2EDuration="1.688346712s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.685350285 +0000 UTC m=+3054.858674711" watchObservedRunningTime="2026-10-30 22:28:51.688346712 +0000 UTC m=+3054.861671136"
Oct 30 22:28:51.811745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:51.811580    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podStartSLOduration=1.8115652789999999 podStartE2EDuration="1.811565279s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:51.809772613 +0000 UTC m=+3054.983097038" watchObservedRunningTime="2026-10-30 22:28:51.811565279 +0000 UTC m=+3054.984889701"
Oct 30 22:28:52.495762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.495710    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:52.496562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.496532    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:52.496713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.496612    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:52.546610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.538392    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:52.546610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.538537    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:52.546610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:52.538608    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:53.496113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:53.496079    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:53.496504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:53.496129    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:53.553620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:53.547040    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:53.553620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:53.547104    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:54.369612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.368369    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:28:54.381813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.381787    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.406559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.406186    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.406559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.406339    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8jxf\" (UniqueName: \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.423946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.423915    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:28:54.479686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.479656    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-5e884c\""
Oct 30 22:28:54.480115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.480097    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-5e884c-dockercfg-zkw6b\""
Oct 30 22:28:54.495924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.495888    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:54.496546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.496463    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:54.508636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.508282    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8jxf\" (UniqueName: \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.509024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.508996    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.509739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.509715    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.538817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.538779    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:54.561715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.561669    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:54.585321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.585287    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8jxf\" (UniqueName: \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\") pod \"payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:54.696823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:54.696729    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:55.117296 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:55.116900    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8783e61f_16a1_404a_a299_0abd5e67cfce.slice/crio-7c366ec116d40e95b494761718713da216e6c7b27870d4d59b05d08b54ef2af5 WatchSource:0}: Error finding container 7c366ec116d40e95b494761718713da216e6c7b27870d4d59b05d08b54ef2af5: Status 404 returned error can't find the container with id 7c366ec116d40e95b494761718713da216e6c7b27870d4d59b05d08b54ef2af5
Oct 30 22:28:55.122253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.122116    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:28:55.473671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.466878    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:28:55.480277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.480183    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.495685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.495660    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:55.495811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.495709    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:55.522658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522616    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522689    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522732    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522786    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522878    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.522998    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.523050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn2ss\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.523162    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.523250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.523202    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.581893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.581842    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:55.582224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.582191    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:55.591518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.591486    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:28:55.609667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.609625    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" event={"ID":"8783e61f-16a1-404a-a299-0abd5e67cfce","Type":"ContainerStarted","Data":"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"}
Oct 30 22:28:55.609832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.609701    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" event={"ID":"8783e61f-16a1-404a-a299-0abd5e67cfce","Type":"ContainerStarted","Data":"7c366ec116d40e95b494761718713da216e6c7b27870d4d59b05d08b54ef2af5"}
Oct 30 22:28:55.610573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.610552    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:28:55.624219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624120    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624238    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624267    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn2ss\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624353    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624397    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624475    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624518    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624557    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.624914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.624823    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.648567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.625061    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.652824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.648822    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.652824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.649933    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.656609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.653243    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.678305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.677337    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.680866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.680841    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.686649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.686574    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.686761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.686738    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn2ss\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:55.777071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.776942    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" podStartSLOduration=1.776920316 podStartE2EDuration="1.776920316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:55.763683878 +0000 UTC m=+3058.937008286" watchObservedRunningTime="2026-10-30 22:28:55.776920316 +0000 UTC m=+3058.950244743"
Oct 30 22:28:55.794861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:55.794619    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:56.280832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.280798    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:28:56.281804 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:56.281774    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc329956_0457_467a_a679_7ee12ae8fcd8.slice/crio-5048fa92a5528b669f72d25ba2fa874a611a772b41147a1de22130869cf396b2 WatchSource:0}: Error finding container 5048fa92a5528b669f72d25ba2fa874a611a772b41147a1de22130869cf396b2: Status 404 returned error can't find the container with id 5048fa92a5528b669f72d25ba2fa874a611a772b41147a1de22130869cf396b2
Oct 30 22:28:56.285747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.285714    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:56.285824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.285788    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:56.285824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.285820    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:56.367621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.366975    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:28:56.383660 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.382015    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.394991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.394718    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:28:56.479613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.478032    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-9a5623-openshift-default-dockercfg-gltcf\""
Oct 30 22:28:56.488971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.488939    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.488989    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489031    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489065    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489110    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489151    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489187    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvxw8\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489213    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.489279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.489270    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.495566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.495537    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:56.495705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.495624    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:56.540082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.539985    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:56.540671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.540635    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:56.590143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590101    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590212    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590262    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590296    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590327    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590369    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590399    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvxw8\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590433    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.590505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590465    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.591359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.590982    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.591359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.591054    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.591359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.591153    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.591359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.591311    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.593144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.592891    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.593144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.593111    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.618769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.618737    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" event={"ID":"dc329956-0457-467a-a679-7ee12ae8fcd8","Type":"ContainerStarted","Data":"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"}
Oct 30 22:28:56.618926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.618788    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" event={"ID":"dc329956-0457-467a-a679-7ee12ae8fcd8","Type":"ContainerStarted","Data":"5048fa92a5528b669f72d25ba2fa874a611a772b41147a1de22130869cf396b2"}
Oct 30 22:28:56.668617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.667888    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvxw8\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.672609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.670627    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\") pod \"e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") " pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.706904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.706870    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:56.774613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.752632    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podStartSLOduration=1.752609096 podStartE2EDuration="1.752609096s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:56.741448981 +0000 UTC m=+3059.914773408" watchObservedRunningTime="2026-10-30 22:28:56.752609096 +0000 UTC m=+3059.925933520"
Oct 30 22:28:56.795671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:56.794723    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:28:57.210510 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:28:57.210471    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ab871ca_51ca_4f5f_8069_82c09968709e.slice/crio-001643ede1d1ade5299c25eff356773cbead2f64a1d9629a8048af4d3040879a WatchSource:0}: Error finding container 001643ede1d1ade5299c25eff356773cbead2f64a1d9629a8048af4d3040879a: Status 404 returned error can't find the container with id 001643ede1d1ade5299c25eff356773cbead2f64a1d9629a8048af4d3040879a
Oct 30 22:28:57.213643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.213600    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:57.213774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.213719    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:57.213838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.213791    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:28:57.217099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.217066    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:28:57.495881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.495787    4882 patch_prober.go:29] interesting pod/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused" start-of-body=
Oct 30 22:28:57.495881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.495857    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.253:15021/healthz/ready\": dial tcp 10.132.0.253:15021: connect: connection refused"
Oct 30 22:28:57.540050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.540011    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused" start-of-body=
Oct 30 22:28:57.540292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.540264    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.254:15021/healthz/ready\": dial tcp 10.132.0.254:15021: connect: connection refused"
Oct 30 22:28:57.638948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.636698    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" event={"ID":"7ab871ca-51ca-4f5f-8069-82c09968709e","Type":"ContainerStarted","Data":"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"}
Oct 30 22:28:57.638948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.636764    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" event={"ID":"7ab871ca-51ca-4f5f-8069-82c09968709e","Type":"ContainerStarted","Data":"001643ede1d1ade5299c25eff356773cbead2f64a1d9629a8048af4d3040879a"}
Oct 30 22:28:57.707549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.707517    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:28:57.797737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.797648    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:28:57.797737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:57.797704    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:28:58.710612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.709702    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:28:58.710612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.709761    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:28:58.807612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.805952    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:28:58.807612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.806012    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:28:58.943610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.943374    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:58.967666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.959819    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:58.997607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:58.997192    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:28:59.024611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.024080    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:59.042773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.042676    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:59.046918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.046848    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:28:59.088937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.088871    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podStartSLOduration=3.088847168 podStartE2EDuration="3.088847168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:28:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:28:57.766897056 +0000 UTC m=+3060.940221524" watchObservedRunningTime="2026-10-30 22:28:59.088847168 +0000 UTC m=+3062.262171595"
Oct 30 22:28:59.117665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.117640    4882 scope.go:122] "RemoveContainer" containerID="5c3f509a5814403b5feef10016a601e68ea98d8f78e17b5672a814ea60228635"
Oct 30 22:28:59.707816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.707782    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:28:59.707997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.707833    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:28:59.796614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.795211    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:28:59.796614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:28:59.795270    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:29:00.707991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:00.707955    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:29:00.708204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:00.708007    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:29:00.795096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:00.795060    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:29:00.795285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:00.795116    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:29:01.707613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:01.707361    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:29:01.707613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:01.707440    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:29:01.795938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:01.795888    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:29:01.796132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:01.795951    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:29:02.078369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:02.078284    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:29:02.707310 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:02.707270    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:29:02.707495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:02.707327    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:29:02.797608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:02.797486    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:29:02.797608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:02.797549    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:29:03.708611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.708112    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:29:03.708611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.708170    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:29:03.795272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.795223    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused" start-of-body=
Oct 30 22:29:03.795560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.795514    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.0:15021/healthz/ready\": dial tcp 10.132.1.0:15021: connect: connection refused"
Oct 30 22:29:03.865062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.865023    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"]
Oct 30 22:29:03.878875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:03.878822    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-a-f7c488-c6j2l"]
Oct 30 22:29:04.710716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:04.710669    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused" start-of-body=
Oct 30 22:29:04.710929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:04.710747    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": dial tcp 10.132.1.1:15021: connect: connection refused"
Oct 30 22:29:04.951775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:04.951730    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39ae35fb-020b-4b88-8107-4321b7fd66a1" path="/var/lib/kubelet/pods/39ae35fb-020b-4b88-8107-4321b7fd66a1/volumes"
Oct 30 22:29:05.140155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:05.139175    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:29:05.140155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:05.140074    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:29:05.141307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:05.141281    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:29:06.779641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:06.757765    4882 patch_prober.go:29] interesting pod/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Oct 30 22:29:06.779641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:06.757833    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.1:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:29:06.952340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:06.952309    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:29:06.952697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:06.952640    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:29:06.954039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:06.954010    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:29:07.641569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:07.641540    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:29:09.193718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:09.193683    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:29:16.894472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:16.894435    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:29:16.900375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:16.900349    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:16.954809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:16.954777    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:29:17.000089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.000057    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-6fc60c\""
Oct 30 22:29:17.030339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.030296    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42788\" (UniqueName: \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.030752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.030721    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.132575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.132533    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42788\" (UniqueName: \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.133054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.133028    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.134187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.134162    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.194445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.194357    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42788\" (UniqueName: \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\") pod \"payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.214356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.214326    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:17.588303 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:29:17.588267    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d9d3c53_392f_47b5_a0ea_ba090268ed24.slice/crio-8ffb837dac1d73a7d14c53486c536f1697c571ea9f67249c4bc3c95546dab250 WatchSource:0}: Error finding container 8ffb837dac1d73a7d14c53486c536f1697c571ea9f67249c4bc3c95546dab250: Status 404 returned error can't find the container with id 8ffb837dac1d73a7d14c53486c536f1697c571ea9f67249c4bc3c95546dab250
Oct 30 22:29:17.598387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.597869    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:29:17.956756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.956667    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" event={"ID":"4d9d3c53-392f-47b5-a0ea-ba090268ed24","Type":"ContainerStarted","Data":"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"}
Oct 30 22:29:17.956756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.956728    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" event={"ID":"4d9d3c53-392f-47b5-a0ea-ba090268ed24","Type":"ContainerStarted","Data":"8ffb837dac1d73a7d14c53486c536f1697c571ea9f67249c4bc3c95546dab250"}
Oct 30 22:29:17.957155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:17.956875    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:18.076067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:18.076004    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" podStartSLOduration=2.075988965 podStartE2EDuration="2.075988965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:18.075268707 +0000 UTC m=+3081.248593134" watchObservedRunningTime="2026-10-30 22:29:18.075988965 +0000 UTC m=+3081.249313402"
Oct 30 22:29:20.810039 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:29:20.809982    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:29:24.217960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.217846    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" podUID="d865258e-33f3-4161-8413-8887845f19b4" containerName="main" containerID="cri-o://7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9" gracePeriod=45
Oct 30 22:29:24.486911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.486810    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:29:24.515152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.515111    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:29:24.515729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.515712    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d865258e-33f3-4161-8413-8887845f19b4" containerName="main"
Oct 30 22:29:24.515841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.515731    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d865258e-33f3-4161-8413-8887845f19b4" containerName="main"
Oct 30 22:29:24.515947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.515934    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d865258e-33f3-4161-8413-8887845f19b4" containerName="main"
Oct 30 22:29:24.519495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.519478    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.577038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.577006    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:29:24.590631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.590572    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-ef00a4\""
Oct 30 22:29:24.590811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.590572    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-ef00a4-dockercfg-8kvb5\""
Oct 30 22:29:24.634995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.634957    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\" (UniqueName: \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635065    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635103    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635184    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635262    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635314    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\") pod \"d865258e-33f3-4161-8413-8887845f19b4\" (UID: \"d865258e-33f3-4161-8413-8887845f19b4\") "
Oct 30 22:29:24.635665 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635629    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.635751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.635729    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrtrv\" (UniqueName: \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.638805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.637102    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:24.638805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.637171    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:24.639140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.639112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:24.640163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.640129    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:29:24.656918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.656758    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "kube-api-access-hvpn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:24.657578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.657550    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm" pod "d865258e-33f3-4161-8413-8887845f19b4" (UID: "d865258e-33f3-4161-8413-8887845f19b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:24.676722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.676081    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:29:24.684402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.684369    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:24.737920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.737816    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrtrv\" (UniqueName: \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.738479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738441    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738564    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738598    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738616    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvpn6\" (UniqueName: \"kubernetes.io/projected/d865258e-33f3-4161-8413-8887845f19b4-kube-api-access-hvpn6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738635    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738651    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d865258e-33f3-4161-8413-8887845f19b4-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.738676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.738664    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d865258e-33f3-4161-8413-8887845f19b4-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:24.740307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.740278    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.743841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.743811    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:29:24.800860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.800826    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrtrv\" (UniqueName: \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\") pod \"payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.833369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.833332    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:24.839219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.839192    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:24.839391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.839368    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5kbcv\" (UniqueName: \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:24.940788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.940747    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:24.941043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.940897    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5kbcv\" (UniqueName: \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:24.941774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:24.941745    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:25.008821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.008744    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5kbcv\" (UniqueName: \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\") pod \"payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:25.009703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.009664    4882 generic.go:356] "Generic (PLEG): container finished" podID="d865258e-33f3-4161-8413-8887845f19b4" containerID="7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9" exitCode=0
Oct 30 22:29:25.009844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.009715    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"
Oct 30 22:29:25.009844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.009755    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" event={"ID":"d865258e-33f3-4161-8413-8887845f19b4","Type":"ContainerDied","Data":"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"}
Oct 30 22:29:25.009844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.009807    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk" event={"ID":"d865258e-33f3-4161-8413-8887845f19b4","Type":"ContainerDied","Data":"e6c6ba0ec61f0ed07fef812263102c51a6eb8a6bfa97f1f0c07b70a86897c734"}
Oct 30 22:29:25.009844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.009830    4882 scope.go:122] "RemoveContainer" containerID="7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"
Oct 30 22:29:25.020438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.020411    4882 scope.go:122] "RemoveContainer" containerID="7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"
Oct 30 22:29:25.020932 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:29:25.020901    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9\": container with ID starting with 7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9 not found: ID does not exist" containerID="7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"
Oct 30 22:29:25.021047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.020938    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9"} err="failed to get container status \"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9\": rpc error: code = NotFound desc = could not find container \"7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9\": container with ID starting with 7e40f59a1d362806af7e0266909944612c972649a1eabd24863a1e2fd9637dd9 not found: ID does not exist"
Oct 30 22:29:25.179494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.179456    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:29:25.183418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.183396    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-praxis-a-f7c488/praxis-route-f7c488-kserve-749d47d9d9-jvrhk"]
Oct 30 22:29:25.233258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.233223    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:29:25.248836 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:29:25.248805    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1202b53e_b96a_4525_a84f_260a0b31c8d0.slice/crio-c64dbea6f9d92d9df799a835d2131072e1ed51b51c51948b1048c10ce00df868 WatchSource:0}: Error finding container c64dbea6f9d92d9df799a835d2131072e1ed51b51c51948b1048c10ce00df868: Status 404 returned error can't find the container with id c64dbea6f9d92d9df799a835d2131072e1ed51b51c51948b1048c10ce00df868
Oct 30 22:29:25.308505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.308475    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:25.695819 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:29:25.695786    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode017891f_d2b9_41ad_9a23_ca0b6c26dbed.slice/crio-ce5402c716ca691dfa49037ec5a2bf0597e15f808b8bdcce7e8f1687fa261a17 WatchSource:0}: Error finding container ce5402c716ca691dfa49037ec5a2bf0597e15f808b8bdcce7e8f1687fa261a17: Status 404 returned error can't find the container with id ce5402c716ca691dfa49037ec5a2bf0597e15f808b8bdcce7e8f1687fa261a17
Oct 30 22:29:25.699623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.699599    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:29:25.709883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.709849    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:29:25.715556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.715538    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.804983 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.804945    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:29:25.851597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851614    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851715    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851747    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851849    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.851963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851946    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmd6v\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.852022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.851971    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.852092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.852065    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953364    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953429    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953483    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953509    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953551    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmd6v\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953577    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953644    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.953839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.954136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.953871    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.954136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.954066    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.954221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.954196    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.954325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.954305    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.955878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.955856    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:25.956126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:25.956107    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:26.018656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.018621    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" event={"ID":"e017891f-d2b9-41ad-9a23-ca0b6c26dbed","Type":"ContainerStarted","Data":"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"}
Oct 30 22:29:26.018851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.018679    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" event={"ID":"e017891f-d2b9-41ad-9a23-ca0b6c26dbed","Type":"ContainerStarted","Data":"ce5402c716ca691dfa49037ec5a2bf0597e15f808b8bdcce7e8f1687fa261a17"}
Oct 30 22:29:26.018851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.018717    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:26.021268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.021237    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmd6v\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:26.021763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.021736    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" event={"ID":"1202b53e-b96a-4525-a84f-260a0b31c8d0","Type":"ContainerStarted","Data":"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"}
Oct 30 22:29:26.021884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.021786    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:26.021884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.021802    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" event={"ID":"1202b53e-b96a-4525-a84f-260a0b31c8d0","Type":"ContainerStarted","Data":"c64dbea6f9d92d9df799a835d2131072e1ed51b51c51948b1048c10ce00df868"}
Oct 30 22:29:26.021994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.021882    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\") pod \"e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") " pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:26.095439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.095370    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:26.142051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.141972    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" podStartSLOduration=2.141947514 podStartE2EDuration="2.141947514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:26.138210971 +0000 UTC m=+3089.311535397" watchObservedRunningTime="2026-10-30 22:29:26.141947514 +0000 UTC m=+3089.315272043"
Oct 30 22:29:26.262330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.262211    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" podStartSLOduration=2.262195844 podStartE2EDuration="2.262195844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:26.257405062 +0000 UTC m=+3089.430729489" watchObservedRunningTime="2026-10-30 22:29:26.262195844 +0000 UTC m=+3089.435520299"
Oct 30 22:29:26.478008 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:29:26.477972    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod803c8a38_8f24_4050_bc51_09f62fb19ddc.slice/crio-caebb401719a79f965e84b586c6a3520d43e6cfde4496494fd3e4511f5eade69 WatchSource:0}: Error finding container caebb401719a79f965e84b586c6a3520d43e6cfde4496494fd3e4511f5eade69: Status 404 returned error can't find the container with id caebb401719a79f965e84b586c6a3520d43e6cfde4496494fd3e4511f5eade69
Oct 30 22:29:26.480684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.480649    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:29:26.480807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.480717    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:29:26.480807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.480745    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:29:26.482230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.482203    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:29:26.944555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:26.944521    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d865258e-33f3-4161-8413-8887845f19b4" path="/var/lib/kubelet/pods/d865258e-33f3-4161-8413-8887845f19b4/volumes"
Oct 30 22:29:27.032150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.032110    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" event={"ID":"803c8a38-8f24-4050-bc51-09f62fb19ddc","Type":"ContainerStarted","Data":"3fdb9f7be0cf9d404c91ba0fccdf3050103422ebf44ea1197eaf001052f76021"}
Oct 30 22:29:27.032336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.032179    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" event={"ID":"803c8a38-8f24-4050-bc51-09f62fb19ddc","Type":"ContainerStarted","Data":"caebb401719a79f965e84b586c6a3520d43e6cfde4496494fd3e4511f5eade69"}
Oct 30 22:29:27.096020 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.095981    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:27.097733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.097702    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:27.097864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.097756    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:27.207317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:27.207202    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podStartSLOduration=2.207185798 podStartE2EDuration="2.207185798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:29:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:29:27.206357291 +0000 UTC m=+3090.379681718" watchObservedRunningTime="2026-10-30 22:29:27.207185798 +0000 UTC m=+3090.380510224"
Oct 30 22:29:28.096704 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:28.096665    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:28.097177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:28.096723    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:28.966702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:28.966670    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:29:29.095865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:29.095833    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:29.096046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:29.095881    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:30.096607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:30.096008    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:30.096607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:30.096065    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:31.096820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:31.096790    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:31.097285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:31.097259    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:32.095894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:32.095857    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:32.096211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:32.096166    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:33.096073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:33.096034    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused" start-of-body=
Oct 30 22:29:33.096720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:33.096686    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.5:15021/healthz/ready\": dial tcp 10.132.1.5:15021: connect: connection refused"
Oct 30 22:29:34.537802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:34.537765    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:34.538226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:34.538180    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:34.539119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:34.539098    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:29:37.035076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:37.035047    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:29:37.035465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:37.035121    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:29:46.714101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:46.714062    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:29:50.813444 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:29:50.813402    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:29:53.054311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:53.054278    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"]
Oct 30 22:29:53.103573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:53.103537    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0a2cf186-dfbbq"]
Oct 30 22:29:54.102861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:54.102824    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:29:54.103327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:54.103083    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy" containerID="cri-o://cc5651fb456f01bd353865e54887a300ab417eb210cd9ddad58662011eeced6a" gracePeriod=30
Oct 30 22:29:54.943529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:54.943489    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7fb113cc-25c2-44d0-afb9-db0c491b3897" path="/var/lib/kubelet/pods/7fb113cc-25c2-44d0-afb9-db0c491b3897/volumes"
Oct 30 22:29:59.217236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.217204    4882 scope.go:122] "RemoveContainer" containerID="38ddc3447410386a7cf3c13e67c591a2887d2c3c6bbbdfb9db038a93d58472d4"
Oct 30 22:29:59.329232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.329199    4882 generic.go:356] "Generic (PLEG): container finished" podID="4492fb74-46d3-488b-8ebe-229689150d0c" containerID="cc5651fb456f01bd353865e54887a300ab417eb210cd9ddad58662011eeced6a" exitCode=0
Oct 30 22:29:59.329408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.329279    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" event={"ID":"4492fb74-46d3-488b-8ebe-229689150d0c","Type":"ContainerDied","Data":"cc5651fb456f01bd353865e54887a300ab417eb210cd9ddad58662011eeced6a"}
Oct 30 22:29:59.398549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.398523    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:29:59.528355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528316    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528408    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528449    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528479    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528645    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.528792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528735    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.528792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528785    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.529030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.528874    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.529092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529056    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.529152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529112    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\") pod \"4492fb74-46d3-488b-8ebe-229689150d0c\" (UID: \"4492fb74-46d3-488b-8ebe-229689150d0c\") "
Oct 30 22:29:59.529212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:29:59.529671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529521    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.529671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529609    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.529884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529741    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.529884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529761    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.529884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529778    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.529884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529791    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4492fb74-46d3-488b-8ebe-229689150d0c-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.529884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.529805    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.531375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.531348    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "kube-api-access-szj6j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.531375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.531350    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:29:59.531713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.531691    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:29:59.532123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.532104    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token" pod "4492fb74-46d3-488b-8ebe-229689150d0c" (UID: "4492fb74-46d3-488b-8ebe-229689150d0c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:29:59.630709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.630619    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.630709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.630654    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szj6j\" (UniqueName: \"kubernetes.io/projected/4492fb74-46d3-488b-8ebe-229689150d0c-kube-api-access-szj6j\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.630709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.630667    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4492fb74-46d3-488b-8ebe-229689150d0c-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:29:59.630709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:29:59.630676    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4492fb74-46d3-488b-8ebe-229689150d0c-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:00.198514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.198469    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:00.199140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.199121    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy"
Oct 30 22:30:00.199196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.199142    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy"
Oct 30 22:30:00.199242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.199233    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" containerName="istio-proxy"
Oct 30 22:30:00.202993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.202972    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:00.203134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.203105    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.207206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.207177    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:00.207370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.207350    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.211276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.211254    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:00.211420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.211394    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.215519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.215499    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.216607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.216568    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:00.221602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.221567    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.257828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.257801    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:30:00.259360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.258920    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:00.263644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.263615    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.263961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.263937    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:00.269117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.269080    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:00.269262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.269197    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.275019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.274252    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:00.291366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.291336    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:00.298208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.298164    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:00.326923 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.326894    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:00.339578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.339545    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"
Oct 30 22:30:00.339578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.339564    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl" event={"ID":"4492fb74-46d3-488b-8ebe-229689150d0c","Type":"ContainerDied","Data":"8a586faada31e21c659d19bb3f737c7bb97576724f0222498c48ca4a5154b64c"}
Oct 30 22:30:00.339861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.339649    4882 scope.go:122] "RemoveContainer" containerID="cc5651fb456f01bd353865e54887a300ab417eb210cd9ddad58662011eeced6a"
Oct 30 22:30:00.339861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.339659    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s889l\" (UniqueName: \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.339861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.339769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhbwg\" (UniqueName: \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.340282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340246    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.340432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340313    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8vmbc\" (UniqueName: \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.340432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340337    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-snrgc\" (UniqueName: \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.340432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340353    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.340432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340384    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2j8cc\" (UniqueName: \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.340432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340416    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340439    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nrr4\" (UniqueName: \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340467    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340485    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340551    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nmr4\" (UniqueName: \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.340787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.340570    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.359871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.359842    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:00.364511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.364480    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:00.441716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441676    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.441917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441837    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8vmbc\" (UniqueName: \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.441917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441886    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-snrgc\" (UniqueName: \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.441917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441909    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.442046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441932    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2j8cc\" (UniqueName: \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.442046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441965    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.442046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.441993    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nrr4\" (UniqueName: \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.442046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442019    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.442270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442051    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.442270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442112    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.442270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442154    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nmr4\" (UniqueName: \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.442270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442185    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.442491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442293    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s889l\" (UniqueName: \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.442491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442366    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhbwg\" (UniqueName: \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.442628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442502    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.442908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442883    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.442989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442929    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.442989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.442959    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.443074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.443016    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.443190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.443172    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.443254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.443236    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.505384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.505305    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhbwg\" (UniqueName: \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\") pod \"maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.506330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.506294    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s889l\" (UniqueName: \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\") pod \"maas-api-key-cleanup-29889990-m6mxz\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.506528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.506506    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-snrgc\" (UniqueName: \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.506770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.506752    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2j8cc\" (UniqueName: \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.509930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.509902    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nrr4\" (UniqueName: \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.511287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.511258    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nmr4\" (UniqueName: \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\") pod \"maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.511525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.511502    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8vmbc\" (UniqueName: \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.523338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.523309    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:00.531766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.531744    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:00.539122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.539097    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:00.547482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.547457    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:00.556930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.556895    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:00.589083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.587796    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:00.598928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.598889    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:00.934860 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:00.934822    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ffe414f_4c3c_45a6_a841_55e4710fefc9.slice/crio-c9c76ffd3ba411f3400ebe89c0085fc2f773ec055914f820dae8ed4aaec6603a WatchSource:0}: Error finding container c9c76ffd3ba411f3400ebe89c0085fc2f773ec055914f820dae8ed4aaec6603a: Status 404 returned error can't find the container with id c9c76ffd3ba411f3400ebe89c0085fc2f773ec055914f820dae8ed4aaec6603a
Oct 30 22:30:00.944685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.944656    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:00.988257 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:00.988226    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73f7c225_adef_4072_a2de_a105c00d2d32.slice/crio-5b6330514ddad1b6a62269652db7427dc6308157e837e63f38ee67b40d4d385f WatchSource:0}: Error finding container 5b6330514ddad1b6a62269652db7427dc6308157e837e63f38ee67b40d4d385f: Status 404 returned error can't find the container with id 5b6330514ddad1b6a62269652db7427dc6308157e837e63f38ee67b40d4d385f
Oct 30 22:30:00.992140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:00.992109    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:01.000371 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:01.000344    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8edb03c_1bf4_4bac_bbd5_6a7090f57863.slice/crio-169e2d9033b004535c170bd4bbe4b95db08c38c1ab7036bfd9139a46057165cc WatchSource:0}: Error finding container 169e2d9033b004535c170bd4bbe4b95db08c38c1ab7036bfd9139a46057165cc: Status 404 returned error can't find the container with id 169e2d9033b004535c170bd4bbe4b95db08c38c1ab7036bfd9139a46057165cc
Oct 30 22:30:01.001435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.001402    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:01.032789 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:01.032761    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954fe549_2577_47f5_b440_e6032657501c.slice/crio-238843359e3eaaba29e3117be67a8509d9413cda9eb74bc2b71cb08c7b628c43 WatchSource:0}: Error finding container 238843359e3eaaba29e3117be67a8509d9413cda9eb74bc2b71cb08c7b628c43: Status 404 returned error can't find the container with id 238843359e3eaaba29e3117be67a8509d9413cda9eb74bc2b71cb08c7b628c43
Oct 30 22:30:01.033476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.033454    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:01.087506 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:01.081482    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ad13d61_e852_4160_8614_4ac621459382.slice/crio-ac0d538cf1d2ba2cff6564bb67fc31e4faaa5d78d66fa09d20bb5eea92054ee6 WatchSource:0}: Error finding container ac0d538cf1d2ba2cff6564bb67fc31e4faaa5d78d66fa09d20bb5eea92054ee6: Status 404 returned error can't find the container with id ac0d538cf1d2ba2cff6564bb67fc31e4faaa5d78d66fa09d20bb5eea92054ee6
Oct 30 22:30:01.087506 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:01.087420    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44a08ce_d8fe_4f8f_8a9e_569f9b76feaf.slice/crio-48802d668bb9b2936ecf35269faba04b2323a77568f48762e08e610b8735a0e0 WatchSource:0}: Error finding container 48802d668bb9b2936ecf35269faba04b2323a77568f48762e08e610b8735a0e0: Status 404 returned error can't find the container with id 48802d668bb9b2936ecf35269faba04b2323a77568f48762e08e610b8735a0e0
Oct 30 22:30:01.088053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.087671    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:01.091120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.091098    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:01.095472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.095445    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:01.183619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.183566    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:30:01.196892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.196862    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-praxis-a-f7c488-openshift-default-54468fb977-zglgl"]
Oct 30 22:30:01.350381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.350348    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerStarted","Data":"ec2299db4449e5ecabb69b7469bd18dfd1bd497cf228fa2db1997cd3787489bf"}
Oct 30 22:30:01.350825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.350412    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerStarted","Data":"c9c76ffd3ba411f3400ebe89c0085fc2f773ec055914f820dae8ed4aaec6603a"}
Oct 30 22:30:01.352668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.352638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerStarted","Data":"169e2d9033b004535c170bd4bbe4b95db08c38c1ab7036bfd9139a46057165cc"}
Oct 30 22:30:01.353993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.353967    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerStarted","Data":"48802d668bb9b2936ecf35269faba04b2323a77568f48762e08e610b8735a0e0"}
Oct 30 22:30:01.356258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.356236    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerStarted","Data":"ac0d538cf1d2ba2cff6564bb67fc31e4faaa5d78d66fa09d20bb5eea92054ee6"}
Oct 30 22:30:01.357601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.357565    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerStarted","Data":"405dace8295ccde21489248db56d4006fb868ae97996c448024b6a9905293c7a"}
Oct 30 22:30:01.358816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.358793    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerStarted","Data":"238843359e3eaaba29e3117be67a8509d9413cda9eb74bc2b71cb08c7b628c43"}
Oct 30 22:30:01.360236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.360216    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerStarted","Data":"5b6330514ddad1b6a62269652db7427dc6308157e837e63f38ee67b40d4d385f"}
Oct 30 22:30:01.469580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.469529    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" podStartSLOduration=1.469515125 podStartE2EDuration="1.469515125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.467682397 +0000 UTC m=+3124.641006828" watchObservedRunningTime="2026-10-30 22:30:01.469515125 +0000 UTC m=+3124.642839553"
Oct 30 22:30:01.593008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:01.592892    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" podStartSLOduration=1.592872228 podStartE2EDuration="1.592872228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:01.587093359 +0000 UTC m=+3124.760417765" watchObservedRunningTime="2026-10-30 22:30:01.592872228 +0000 UTC m=+3124.766196656"
Oct 30 22:30:02.369176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.369140    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ad13d61-e852-4160-8614-4ac621459382" containerID="2487ce7ec7e6ec55067d0f89d5b6a59b9973afb1e8d6c0fc8e54be6b2b04e03f" exitCode=60
Oct 30 22:30:02.369708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.369218    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerDied","Data":"2487ce7ec7e6ec55067d0f89d5b6a59b9973afb1e8d6c0fc8e54be6b2b04e03f"}
Oct 30 22:30:02.369708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.369492    4882 scope.go:122] "RemoveContainer" containerID="2487ce7ec7e6ec55067d0f89d5b6a59b9973afb1e8d6c0fc8e54be6b2b04e03f"
Oct 30 22:30:02.371116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.371053    4882 generic.go:356] "Generic (PLEG): container finished" podID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerID="bc15404cb67ff2fc4cecc5bcd5e1a9eb6b6436c701ab6e83706765f5de1f2f7d" exitCode=60
Oct 30 22:30:02.371204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.371139    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerDied","Data":"bc15404cb67ff2fc4cecc5bcd5e1a9eb6b6436c701ab6e83706765f5de1f2f7d"}
Oct 30 22:30:02.371353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.371335    4882 scope.go:122] "RemoveContainer" containerID="bc15404cb67ff2fc4cecc5bcd5e1a9eb6b6436c701ab6e83706765f5de1f2f7d"
Oct 30 22:30:02.373068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.373054    4882 generic.go:356] "Generic (PLEG): container finished" podID="954fe549-2577-47f5-b440-e6032657501c" containerID="c42e1fb1e262d240ac1ba1461a2187708c72e7f4924841f206be74839b42093a" exitCode=60
Oct 30 22:30:02.373128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.373116    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerDied","Data":"c42e1fb1e262d240ac1ba1461a2187708c72e7f4924841f206be74839b42093a"}
Oct 30 22:30:02.373399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.373378    4882 scope.go:122] "RemoveContainer" containerID="c42e1fb1e262d240ac1ba1461a2187708c72e7f4924841f206be74839b42093a"
Oct 30 22:30:02.375115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.375096    4882 generic.go:356] "Generic (PLEG): container finished" podID="73f7c225-adef-4072-a2de-a105c00d2d32" containerID="284595767a78e93b4ca2f56ce6bf3038162285606578a7497de0ff79112784b7" exitCode=60
Oct 30 22:30:02.375198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.375175    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerDied","Data":"284595767a78e93b4ca2f56ce6bf3038162285606578a7497de0ff79112784b7"}
Oct 30 22:30:02.375366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.375351    4882 scope.go:122] "RemoveContainer" containerID="284595767a78e93b4ca2f56ce6bf3038162285606578a7497de0ff79112784b7"
Oct 30 22:30:02.377481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.377448    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerID="ec2299db4449e5ecabb69b7469bd18dfd1bd497cf228fa2db1997cd3787489bf" exitCode=60
Oct 30 22:30:02.377607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.377501    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerDied","Data":"ec2299db4449e5ecabb69b7469bd18dfd1bd497cf228fa2db1997cd3787489bf"}
Oct 30 22:30:02.377713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.377699    4882 scope.go:122] "RemoveContainer" containerID="ec2299db4449e5ecabb69b7469bd18dfd1bd497cf228fa2db1997cd3787489bf"
Oct 30 22:30:02.379225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.379200    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerID="f1d011430593fdc05cc0823e1b297986775f08f720ec91103258748bf38f2e73" exitCode=60
Oct 30 22:30:02.379327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.379286    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerDied","Data":"f1d011430593fdc05cc0823e1b297986775f08f720ec91103258748bf38f2e73"}
Oct 30 22:30:02.379530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.379511    4882 scope.go:122] "RemoveContainer" containerID="f1d011430593fdc05cc0823e1b297986775f08f720ec91103258748bf38f2e73"
Oct 30 22:30:02.381407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.381390    4882 generic.go:356] "Generic (PLEG): container finished" podID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerID="5a0de91537141ba65d27eff9f95a31d9f8d273791c6d9e3d1a0bab54e532e111" exitCode=60
Oct 30 22:30:02.381459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.381437    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerDied","Data":"5a0de91537141ba65d27eff9f95a31d9f8d273791c6d9e3d1a0bab54e532e111"}
Oct 30 22:30:02.381667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.381654    4882 scope.go:122] "RemoveContainer" containerID="5a0de91537141ba65d27eff9f95a31d9f8d273791c6d9e3d1a0bab54e532e111"
Oct 30 22:30:02.948393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:02.948364    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4492fb74-46d3-488b-8ebe-229689150d0c" path="/var/lib/kubelet/pods/4492fb74-46d3-488b-8ebe-229689150d0c/volumes"
Oct 30 22:30:03.395631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.395558    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerStarted","Data":"a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"}
Oct 30 22:30:03.397744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.397716    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerStarted","Data":"16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"}
Oct 30 22:30:03.399727 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.399705    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerStarted","Data":"2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"}
Oct 30 22:30:03.401745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.401726    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerStarted","Data":"62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"}
Oct 30 22:30:03.403533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.403511    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerStarted","Data":"49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"}
Oct 30 22:30:03.405433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.405406    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerStarted","Data":"7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"}
Oct 30 22:30:03.407464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.407444    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerStarted","Data":"ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"}
Oct 30 22:30:03.512969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.512905    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" podStartSLOduration=3.5128907700000003 podStartE2EDuration="3.51289077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.51242313 +0000 UTC m=+3126.685747561" watchObservedRunningTime="2026-10-30 22:30:03.51289077 +0000 UTC m=+3126.686215215"
Oct 30 22:30:03.875393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.875319    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" podStartSLOduration=3.875297275 podStartE2EDuration="3.875297275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.868399353 +0000 UTC m=+3127.041723782" watchObservedRunningTime="2026-10-30 22:30:03.875297275 +0000 UTC m=+3127.048621703"
Oct 30 22:30:03.990134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:03.990028    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" podStartSLOduration=3.990010387 podStartE2EDuration="3.990010387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:03.985449511 +0000 UTC m=+3127.158773939" watchObservedRunningTime="2026-10-30 22:30:03.990010387 +0000 UTC m=+3127.163334871"
Oct 30 22:30:04.103116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.103065    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" podStartSLOduration=4.10304998 podStartE2EDuration="4.10304998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.101813456 +0000 UTC m=+3127.275137884" watchObservedRunningTime="2026-10-30 22:30:04.10304998 +0000 UTC m=+3127.276374424"
Oct 30 22:30:04.417335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.417292    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ad13d61-e852-4160-8614-4ac621459382" containerID="a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6" exitCode=60
Oct 30 22:30:04.417883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.417377    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerDied","Data":"a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"}
Oct 30 22:30:04.417883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.417442    4882 scope.go:122] "RemoveContainer" containerID="2487ce7ec7e6ec55067d0f89d5b6a59b9973afb1e8d6c0fc8e54be6b2b04e03f"
Oct 30 22:30:04.417883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.417702    4882 scope.go:122] "RemoveContainer" containerID="a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"
Oct 30 22:30:04.418072 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.418047    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f_odh-ai-gateway-infra(6ad13d61-e852-4160-8614-4ac621459382)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" podUID="6ad13d61-e852-4160-8614-4ac621459382"
Oct 30 22:30:04.419801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.419783    4882 generic.go:356] "Generic (PLEG): container finished" podID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerID="16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344" exitCode=60
Oct 30 22:30:04.419903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.419852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerDied","Data":"16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"}
Oct 30 22:30:04.420137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.420119    4882 scope.go:122] "RemoveContainer" containerID="16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"
Oct 30 22:30:04.420425 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.420404    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6mxz_odh-ai-gateway-infra(9d579993-8868-4fe9-b437-43fcaa92ddb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9"
Oct 30 22:30:04.421845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.421825    4882 generic.go:356] "Generic (PLEG): container finished" podID="954fe549-2577-47f5-b440-e6032657501c" containerID="2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba" exitCode=60
Oct 30 22:30:04.421957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.421895    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerDied","Data":"2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"}
Oct 30 22:30:04.422228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.422210    4882 scope.go:122] "RemoveContainer" containerID="2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"
Oct 30 22:30:04.422548 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.422519    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf_odh-ai-gateway-infra(954fe549-2577-47f5-b440-e6032657501c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" podUID="954fe549-2577-47f5-b440-e6032657501c"
Oct 30 22:30:04.424019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.423997    4882 generic.go:356] "Generic (PLEG): container finished" podID="73f7c225-adef-4072-a2de-a105c00d2d32" containerID="62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e" exitCode=60
Oct 30 22:30:04.424124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.424083    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerDied","Data":"62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"}
Oct 30 22:30:04.424314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.424300    4882 scope.go:122] "RemoveContainer" containerID="62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"
Oct 30 22:30:04.424638 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.424612    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45_odh-ai-gateway-infra(73f7c225-adef-4072-a2de-a105c00d2d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" podUID="73f7c225-adef-4072-a2de-a105c00d2d32"
Oct 30 22:30:04.426189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.426164    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerID="49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e" exitCode=60
Oct 30 22:30:04.426292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.426249    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerDied","Data":"49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"}
Oct 30 22:30:04.426478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.426462    4882 scope.go:122] "RemoveContainer" containerID="49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"
Oct 30 22:30:04.426750 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.426725    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb_odh-ai-gateway-infra(1ffe414f-4c3c-45a6-a841-55e4710fefc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9"
Oct 30 22:30:04.428346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.428328    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerID="7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833" exitCode=60
Oct 30 22:30:04.428452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.428410    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerDied","Data":"7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"}
Oct 30 22:30:04.428733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.428715    4882 scope.go:122] "RemoveContainer" containerID="7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"
Oct 30 22:30:04.428991 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.428972    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn_odh-ai-gateway-infra(a8edb03c-1bf4-4bac-bbd5-6a7090f57863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863"
Oct 30 22:30:04.430494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.430469    4882 generic.go:356] "Generic (PLEG): container finished" podID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerID="ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d" exitCode=60
Oct 30 22:30:04.430634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.430543    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerDied","Data":"ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"}
Oct 30 22:30:04.430784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.430769    4882 scope.go:122] "RemoveContainer" containerID="ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"
Oct 30 22:30:04.431021 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:04.431001    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7_odh-ai-gateway-infra(c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf"
Oct 30 22:30:04.433105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.433090    4882 scope.go:122] "RemoveContainer" containerID="bc15404cb67ff2fc4cecc5bcd5e1a9eb6b6436c701ab6e83706765f5de1f2f7d"
Oct 30 22:30:04.448014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.447990    4882 scope.go:122] "RemoveContainer" containerID="c42e1fb1e262d240ac1ba1461a2187708c72e7f4924841f206be74839b42093a"
Oct 30 22:30:04.462749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.462712    4882 scope.go:122] "RemoveContainer" containerID="284595767a78e93b4ca2f56ce6bf3038162285606578a7497de0ff79112784b7"
Oct 30 22:30:04.477225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.477203    4882 scope.go:122] "RemoveContainer" containerID="ec2299db4449e5ecabb69b7469bd18dfd1bd497cf228fa2db1997cd3787489bf"
Oct 30 22:30:04.491410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.491388    4882 scope.go:122] "RemoveContainer" containerID="f1d011430593fdc05cc0823e1b297986775f08f720ec91103258748bf38f2e73"
Oct 30 22:30:04.506285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.506259    4882 scope.go:122] "RemoveContainer" containerID="5a0de91537141ba65d27eff9f95a31d9f8d273791c6d9e3d1a0bab54e532e111"
Oct 30 22:30:04.540142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:04.540089    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" podStartSLOduration=4.540074005 podStartE2EDuration="4.540074005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:04.219850416 +0000 UTC m=+3127.393174840" watchObservedRunningTime="2026-10-30 22:30:04.540074005 +0000 UTC m=+3127.713398431"
Oct 30 22:30:05.441059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.441034    4882 scope.go:122] "RemoveContainer" containerID="ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"
Oct 30 22:30:05.441487 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.441343    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7_odh-ai-gateway-infra(c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf"
Oct 30 22:30:05.442814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.442790    4882 scope.go:122] "RemoveContainer" containerID="a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"
Oct 30 22:30:05.443107 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.443081    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f_odh-ai-gateway-infra(6ad13d61-e852-4160-8614-4ac621459382)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" podUID="6ad13d61-e852-4160-8614-4ac621459382"
Oct 30 22:30:05.444388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.444365    4882 scope.go:122] "RemoveContainer" containerID="16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"
Oct 30 22:30:05.444639 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.444612    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6mxz_odh-ai-gateway-infra(9d579993-8868-4fe9-b437-43fcaa92ddb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9"
Oct 30 22:30:05.446019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.446000    4882 scope.go:122] "RemoveContainer" containerID="2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"
Oct 30 22:30:05.446259 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.446235    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf_odh-ai-gateway-infra(954fe549-2577-47f5-b440-e6032657501c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" podUID="954fe549-2577-47f5-b440-e6032657501c"
Oct 30 22:30:05.447579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.447563    4882 scope.go:122] "RemoveContainer" containerID="62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"
Oct 30 22:30:05.447804 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.447782    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45_odh-ai-gateway-infra(73f7c225-adef-4072-a2de-a105c00d2d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" podUID="73f7c225-adef-4072-a2de-a105c00d2d32"
Oct 30 22:30:05.449132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.449117    4882 scope.go:122] "RemoveContainer" containerID="49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"
Oct 30 22:30:05.449295 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.449278    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb_odh-ai-gateway-infra(1ffe414f-4c3c-45a6-a841-55e4710fefc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9"
Oct 30 22:30:05.450714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:05.450695    4882 scope.go:122] "RemoveContainer" containerID="7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"
Oct 30 22:30:05.450913 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:05.450894    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn_odh-ai-gateway-infra(a8edb03c-1bf4-4bac-bbd5-6a7090f57863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863"
Oct 30 22:30:09.781357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.781322    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"]
Oct 30 22:30:09.787364 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.787337    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:09.838829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.838793    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"]
Oct 30 22:30:09.883461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.883429    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:30:09.968309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.968270    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:09.968498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.968470    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bskvm\" (UniqueName: \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:09.968541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:09.968503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.069786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.069694    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bskvm\" (UniqueName: \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.069786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.069743    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.069786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.069787    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.070527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.070500    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.072353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.072326    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.131843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.131803    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bskvm\" (UniqueName: \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\") pod \"maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.402488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.402449    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:10.784354 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:10.784319    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod651f12ba_bf0d_456f_987c_8a91a9caabe1.slice/crio-ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030 WatchSource:0}: Error finding container ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030: Status 404 returned error can't find the container with id ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030
Oct 30 22:30:10.788431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:10.788400    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"]
Oct 30 22:30:11.051828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:11.051786    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"]
Oct 30 22:30:11.056190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:11.056152    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a851f667-rmnrf"]
Oct 30 22:30:11.501633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:11.501597    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv" event={"ID":"651f12ba-bf0d-456f-987c-8a91a9caabe1","Type":"ContainerStarted","Data":"5c100462724b076ac023de09a4f3c46ee5b9bbadfa1907f7f3af58b90c84b278"}
Oct 30 22:30:11.501883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:11.501668    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv" event={"ID":"651f12ba-bf0d-456f-987c-8a91a9caabe1","Type":"ContainerStarted","Data":"ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030"}
Oct 30 22:30:11.620096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:11.620040    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv" podStartSLOduration=2.620023977 podStartE2EDuration="2.620023977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:11.619616701 +0000 UTC m=+3134.792941125" watchObservedRunningTime="2026-10-30 22:30:11.620023977 +0000 UTC m=+3134.793348405"
Oct 30 22:30:12.511152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:12.511118    4882 generic.go:356] "Generic (PLEG): container finished" podID="651f12ba-bf0d-456f-987c-8a91a9caabe1" containerID="5c100462724b076ac023de09a4f3c46ee5b9bbadfa1907f7f3af58b90c84b278" exitCode=0
Oct 30 22:30:12.511554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:12.511206    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv" event={"ID":"651f12ba-bf0d-456f-987c-8a91a9caabe1","Type":"ContainerDied","Data":"5c100462724b076ac023de09a4f3c46ee5b9bbadfa1907f7f3af58b90c84b278"}
Oct 30 22:30:12.943921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:12.943880    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="163d0767-b2d5-43f6-a29d-ee9e38991da1" path="/var/lib/kubelet/pods/163d0767-b2d5-43f6-a29d-ee9e38991da1/volumes"
Oct 30 22:30:13.676537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.676503    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:13.715745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.715714    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\") pod \"651f12ba-bf0d-456f-987c-8a91a9caabe1\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") "
Oct 30 22:30:13.715745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.715749    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\" (UniqueName: \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\") pod \"651f12ba-bf0d-456f-987c-8a91a9caabe1\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") "
Oct 30 22:30:13.715966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.715805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\") pod \"651f12ba-bf0d-456f-987c-8a91a9caabe1\" (UID: \"651f12ba-bf0d-456f-987c-8a91a9caabe1\") "
Oct 30 22:30:13.716098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.716073    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca" pod "651f12ba-bf0d-456f-987c-8a91a9caabe1" (UID: "651f12ba-bf0d-456f-987c-8a91a9caabe1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:13.716313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.716292    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.717757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.717733    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm" pod "651f12ba-bf0d-456f-987c-8a91a9caabe1" (UID: "651f12ba-bf0d-456f-987c-8a91a9caabe1"). InnerVolumeSpecName "kube-api-access-bskvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:13.717850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.717832    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca" pod "651f12ba-bf0d-456f-987c-8a91a9caabe1" (UID: "651f12ba-bf0d-456f-987c-8a91a9caabe1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:30:13.818663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.818016    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/651f12ba-bf0d-456f-987c-8a91a9caabe1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.818663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.818059    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bskvm\" (UniqueName: \"kubernetes.io/projected/651f12ba-bf0d-456f-987c-8a91a9caabe1-kube-api-access-bskvm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:13.936173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.936139    4882 scope.go:122] "RemoveContainer" containerID="2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"
Oct 30 22:30:13.936415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:13.936217    4882 scope.go:122] "RemoveContainer" containerID="a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"
Oct 30 22:30:14.529650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.529563    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"
Oct 30 22:30:14.529650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.529606    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv" event={"ID":"651f12ba-bf0d-456f-987c-8a91a9caabe1","Type":"ContainerDied","Data":"ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030"}
Oct 30 22:30:14.529846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.529685    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ab185865d775cbf7f7fee5294e1be07c7f5bd8c28a917007f6d0b65763db7030"
Oct 30 22:30:14.531860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.531833    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerStarted","Data":"a9eb03979d624a310b7a5c311f39479e0e8fef6eb6fac579b0277679c3dd47e4"}
Oct 30 22:30:14.533994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.533968    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerStarted","Data":"41984e9825d9ef4e8d12e146f5b41a6072a58612272b81c6b589599e8ca26b73"}
Oct 30 22:30:14.936543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.936509    4882 scope.go:122] "RemoveContainer" containerID="7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"
Oct 30 22:30:14.937001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:14.936739    4882 scope.go:122] "RemoveContainer" containerID="ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"
Oct 30 22:30:15.543551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.543465    4882 generic.go:356] "Generic (PLEG): container finished" podID="6ad13d61-e852-4160-8614-4ac621459382" containerID="a9eb03979d624a310b7a5c311f39479e0e8fef6eb6fac579b0277679c3dd47e4" exitCode=60
Oct 30 22:30:15.543551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.543536    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerDied","Data":"a9eb03979d624a310b7a5c311f39479e0e8fef6eb6fac579b0277679c3dd47e4"}
Oct 30 22:30:15.543792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.543634    4882 scope.go:122] "RemoveContainer" containerID="a488b55292490387effaa8f138075da51b9396b128f92ce53f9016466788acb6"
Oct 30 22:30:15.544062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.544035    4882 scope.go:122] "RemoveContainer" containerID="a9eb03979d624a310b7a5c311f39479e0e8fef6eb6fac579b0277679c3dd47e4"
Oct 30 22:30:15.544359 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:15.544335    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f_odh-ai-gateway-infra(6ad13d61-e852-4160-8614-4ac621459382)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" podUID="6ad13d61-e852-4160-8614-4ac621459382"
Oct 30 22:30:15.545915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.545893    4882 generic.go:356] "Generic (PLEG): container finished" podID="954fe549-2577-47f5-b440-e6032657501c" containerID="41984e9825d9ef4e8d12e146f5b41a6072a58612272b81c6b589599e8ca26b73" exitCode=60
Oct 30 22:30:15.546009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.545963    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerDied","Data":"41984e9825d9ef4e8d12e146f5b41a6072a58612272b81c6b589599e8ca26b73"}
Oct 30 22:30:15.546334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.546314    4882 scope.go:122] "RemoveContainer" containerID="41984e9825d9ef4e8d12e146f5b41a6072a58612272b81c6b589599e8ca26b73"
Oct 30 22:30:15.546657 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:15.546635    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf_odh-ai-gateway-infra(954fe549-2577-47f5-b440-e6032657501c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" podUID="954fe549-2577-47f5-b440-e6032657501c"
Oct 30 22:30:15.548158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.548127    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerStarted","Data":"2314f3270d87989027cb5a52493689270244d3c68c36515371056382fd0badfa"}
Oct 30 22:30:15.550274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.550253    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerStarted","Data":"472371eb44076f20deeb5c16e1666252848f5fa51a0ccda43280ab75dc5e8484"}
Oct 30 22:30:15.558176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.558149    4882 scope.go:122] "RemoveContainer" containerID="2a72381b32e30b433fa77f74c8afac5b0cb0772c2274275f067d16da9fd50fba"
Oct 30 22:30:15.667402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.667366    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:15.781737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:15.781699    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:16.564948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.564915    4882 generic.go:356] "Generic (PLEG): container finished" podID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerID="2314f3270d87989027cb5a52493689270244d3c68c36515371056382fd0badfa" exitCode=60
Oct 30 22:30:16.565432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.565019    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerDied","Data":"2314f3270d87989027cb5a52493689270244d3c68c36515371056382fd0badfa"}
Oct 30 22:30:16.565432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.565184    4882 scope.go:122] "RemoveContainer" containerID="7299de557e9b5cb9f79ace474fc68330307fe54e13414810bfa01870c295a833"
Oct 30 22:30:16.565549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.565475    4882 scope.go:122] "RemoveContainer" containerID="2314f3270d87989027cb5a52493689270244d3c68c36515371056382fd0badfa"
Oct 30 22:30:16.566024 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:16.565955    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn_odh-ai-gateway-infra(a8edb03c-1bf4-4bac-bbd5-6a7090f57863)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863"
Oct 30 22:30:16.570269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.569877    4882 generic.go:356] "Generic (PLEG): container finished" podID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerID="472371eb44076f20deeb5c16e1666252848f5fa51a0ccda43280ab75dc5e8484" exitCode=60
Oct 30 22:30:16.570269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.569946    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerDied","Data":"472371eb44076f20deeb5c16e1666252848f5fa51a0ccda43280ab75dc5e8484"}
Oct 30 22:30:16.570481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.570456    4882 scope.go:122] "RemoveContainer" containerID="472371eb44076f20deeb5c16e1666252848f5fa51a0ccda43280ab75dc5e8484"
Oct 30 22:30:16.570932 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:16.570905    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7_odh-ai-gateway-infra(c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf"
Oct 30 22:30:16.580748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.580716    4882 scope.go:122] "RemoveContainer" containerID="ad040fa13c993973ed4a4657e7cc0974154f880452d8f6eb6ce649454899d46d"
Oct 30 22:30:16.768409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.768376    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:16.771148 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.771123    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:16.819415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.819315    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:16.853393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.853352    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\") pod \"6ad13d61-e852-4160-8614-4ac621459382\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") "
Oct 30 22:30:16.853599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.853548    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\" (UniqueName: \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\") pod \"6ad13d61-e852-4160-8614-4ac621459382\" (UID: \"6ad13d61-e852-4160-8614-4ac621459382\") "
Oct 30 22:30:16.853659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.853647    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\" (UniqueName: \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\") pod \"954fe549-2577-47f5-b440-e6032657501c\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") "
Oct 30 22:30:16.853707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.853668    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\") pod \"954fe549-2577-47f5-b440-e6032657501c\" (UID: \"954fe549-2577-47f5-b440-e6032657501c\") "
Oct 30 22:30:16.853793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.853767    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca" pod "6ad13d61-e852-4160-8614-4ac621459382" (UID: "6ad13d61-e852-4160-8614-4ac621459382"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.854082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.854061    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca" pod "954fe549-2577-47f5-b440-e6032657501c" (UID: "954fe549-2577-47f5-b440-e6032657501c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:16.854211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.854192    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/954fe549-2577-47f5-b440-e6032657501c-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.854211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.854208    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ad13d61-e852-4160-8614-4ac621459382-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.855753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.855726    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc" pod "6ad13d61-e852-4160-8614-4ac621459382" (UID: "6ad13d61-e852-4160-8614-4ac621459382"). InnerVolumeSpecName "kube-api-access-2j8cc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.855856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.855806    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg" pod "954fe549-2577-47f5-b440-e6032657501c" (UID: "954fe549-2577-47f5-b440-e6032657501c"). InnerVolumeSpecName "kube-api-access-fhbwg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:16.936396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.936362    4882 scope.go:122] "RemoveContainer" containerID="49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"
Oct 30 22:30:16.936603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.936431    4882 scope.go:122] "RemoveContainer" containerID="16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"
Oct 30 22:30:16.956318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.956274    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhbwg\" (UniqueName: \"kubernetes.io/projected/954fe549-2577-47f5-b440-e6032657501c-kube-api-access-fhbwg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:16.956318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:16.956324    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2j8cc\" (UniqueName: \"kubernetes.io/projected/6ad13d61-e852-4160-8614-4ac621459382-kube-api-access-2j8cc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.051027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.050980    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:17.292573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.292520    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:30:17.585326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.585296    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"
Oct 30 22:30:17.585769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.585302    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f" event={"ID":"6ad13d61-e852-4160-8614-4ac621459382","Type":"ContainerDied","Data":"ac0d538cf1d2ba2cff6564bb67fc31e4faaa5d78d66fa09d20bb5eea92054ee6"}
Oct 30 22:30:17.585769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.585387    4882 scope.go:122] "RemoveContainer" containerID="a9eb03979d624a310b7a5c311f39479e0e8fef6eb6fac579b0277679c3dd47e4"
Oct 30 22:30:17.588190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.588154    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerStarted","Data":"2e857f4db391519d2e8733f8ae708545916b202b1707b5b9834e42d3021ae3d8"}
Oct 30 22:30:17.591421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.591001    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"
Oct 30 22:30:17.591421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.591031    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf" event={"ID":"954fe549-2577-47f5-b440-e6032657501c","Type":"ContainerDied","Data":"238843359e3eaaba29e3117be67a8509d9413cda9eb74bc2b71cb08c7b628c43"}
Oct 30 22:30:17.600940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.599902    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerStarted","Data":"87d06b22c97e6b6971f8f94ae4c5cbcf87f5b25a314fc6fe6daa76ba2fd5bf7c"}
Oct 30 22:30:17.606149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.603033    4882 scope.go:122] "RemoveContainer" containerID="41984e9825d9ef4e8d12e146f5b41a6072a58612272b81c6b589599e8ca26b73"
Oct 30 22:30:17.735746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.735711    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:17.765386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.765349    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\" (UniqueName: \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\") pod \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") "
Oct 30 22:30:17.765568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.765490    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\") pod \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\" (UID: \"a8edb03c-1bf4-4bac-bbd5-6a7090f57863\") "
Oct 30 22:30:17.766139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.765845    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca" pod "a8edb03c-1bf4-4bac-bbd5-6a7090f57863" (UID: "a8edb03c-1bf4-4bac-bbd5-6a7090f57863"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.766139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.766103    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.767839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.767814    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc" pod "a8edb03c-1bf4-4bac-bbd5-6a7090f57863" (UID: "a8edb03c-1bf4-4bac-bbd5-6a7090f57863"). InnerVolumeSpecName "kube-api-access-snrgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.801036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.801000    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:17.867028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.866941    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\" (UniqueName: \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\") pod \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") "
Oct 30 22:30:17.867028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.866999    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\") pod \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\" (UID: \"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf\") "
Oct 30 22:30:17.867361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.867349    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-snrgc\" (UniqueName: \"kubernetes.io/projected/a8edb03c-1bf4-4bac-bbd5-6a7090f57863-kube-api-access-snrgc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.867409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.867378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca" pod "c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" (UID: "c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:17.869205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.869169    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4" pod "c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" (UID: "c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf"). InnerVolumeSpecName "kube-api-access-4nrr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:17.935593 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.935543    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:17.936757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.936736    4882 scope.go:122] "RemoveContainer" containerID="62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"
Oct 30 22:30:17.941677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.941653    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29889990-kb44f"]
Oct 30 22:30:17.969217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.969117    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nrr4\" (UniqueName: \"kubernetes.io/projected/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-kube-api-access-4nrr4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:17.969217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:17.969161    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:18.307507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.307463    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:18.317250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.317210    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-26743a-29889990-j6kjf"]
Oct 30 22:30:18.611471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.611434    4882 generic.go:356] "Generic (PLEG): container finished" podID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerID="2e857f4db391519d2e8733f8ae708545916b202b1707b5b9834e42d3021ae3d8" exitCode=60
Oct 30 22:30:18.612043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.611505    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerDied","Data":"2e857f4db391519d2e8733f8ae708545916b202b1707b5b9834e42d3021ae3d8"}
Oct 30 22:30:18.612043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.611565    4882 scope.go:122] "RemoveContainer" containerID="16733e4c3d9f916738cc718d06a706d55358630d1368d951263910734e316344"
Oct 30 22:30:18.612043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.612013    4882 scope.go:122] "RemoveContainer" containerID="2e857f4db391519d2e8733f8ae708545916b202b1707b5b9834e42d3021ae3d8"
Oct 30 22:30:18.612412 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:18.612381    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889990-m6mxz_odh-ai-gateway-infra(9d579993-8868-4fe9-b437-43fcaa92ddb9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9"
Oct 30 22:30:18.614819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.614794    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerStarted","Data":"b93f594f82bff1f75c4b8fa43423c00e4483d10e58a7ee197f3e64a68e084b4f"}
Oct 30 22:30:18.616813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.616787    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerID="87d06b22c97e6b6971f8f94ae4c5cbcf87f5b25a314fc6fe6daa76ba2fd5bf7c" exitCode=60
Oct 30 22:30:18.616925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.616860    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerDied","Data":"87d06b22c97e6b6971f8f94ae4c5cbcf87f5b25a314fc6fe6daa76ba2fd5bf7c"}
Oct 30 22:30:18.617231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.617214    4882 scope.go:122] "RemoveContainer" containerID="87d06b22c97e6b6971f8f94ae4c5cbcf87f5b25a314fc6fe6daa76ba2fd5bf7c"
Oct 30 22:30:18.617524 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:18.617505    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb_odh-ai-gateway-infra(1ffe414f-4c3c-45a6-a841-55e4710fefc9)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9"
Oct 30 22:30:18.618264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.618251    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"
Oct 30 22:30:18.618345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.618294    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn" event={"ID":"a8edb03c-1bf4-4bac-bbd5-6a7090f57863","Type":"ContainerDied","Data":"169e2d9033b004535c170bd4bbe4b95db08c38c1ab7036bfd9139a46057165cc"}
Oct 30 22:30:18.619767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.619742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7" event={"ID":"c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf","Type":"ContainerDied","Data":"48802d668bb9b2936ecf35269faba04b2323a77568f48762e08e610b8735a0e0"}
Oct 30 22:30:18.619767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.619758    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"
Oct 30 22:30:18.628801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.628707    4882 scope.go:122] "RemoveContainer" containerID="49369cc92d5ccad7456216ffa222dd4323213f92ab9133bf57ea95b9adfa9c9e"
Oct 30 22:30:18.644010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.643988    4882 scope.go:122] "RemoveContainer" containerID="2314f3270d87989027cb5a52493689270244d3c68c36515371056382fd0badfa"
Oct 30 22:30:18.654559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.654533    4882 scope.go:122] "RemoveContainer" containerID="472371eb44076f20deeb5c16e1666252848f5fa51a0ccda43280ab75dc5e8484"
Oct 30 22:30:18.830094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.830059    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:18.954210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.954178    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ad13d61-e852-4160-8614-4ac621459382" path="/var/lib/kubelet/pods/6ad13d61-e852-4160-8614-4ac621459382/volumes"
Oct 30 22:30:18.954851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:18.954835    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954fe549-2577-47f5-b440-e6032657501c" path="/var/lib/kubelet/pods/954fe549-2577-47f5-b440-e6032657501c/volumes"
Oct 30 22:30:19.063651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.063607    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:19.102067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.102030    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:19.116844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.116811    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29889990-wsmpn"]
Oct 30 22:30:19.228445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.228358    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:19.232900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.232871    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29889990-qptk7"]
Oct 30 22:30:19.636471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.636437    4882 generic.go:356] "Generic (PLEG): container finished" podID="73f7c225-adef-4072-a2de-a105c00d2d32" containerID="b93f594f82bff1f75c4b8fa43423c00e4483d10e58a7ee197f3e64a68e084b4f" exitCode=60
Oct 30 22:30:19.636939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.636486    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerDied","Data":"b93f594f82bff1f75c4b8fa43423c00e4483d10e58a7ee197f3e64a68e084b4f"}
Oct 30 22:30:19.636939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.636558    4882 scope.go:122] "RemoveContainer" containerID="62a4e85a6f9f9f88d5857e0daada0bdb9210bc5483470d70136e01b7f267e84e"
Oct 30 22:30:19.636939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.636937    4882 scope.go:122] "RemoveContainer" containerID="b93f594f82bff1f75c4b8fa43423c00e4483d10e58a7ee197f3e64a68e084b4f"
Oct 30 22:30:19.637302 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:19.637280    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45_odh-ai-gateway-infra(73f7c225-adef-4072-a2de-a105c00d2d32)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" podUID="73f7c225-adef-4072-a2de-a105c00d2d32"
Oct 30 22:30:19.824318 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.824290    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:19.864433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.864407    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:19.893282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893235    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\" (UniqueName: \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\") pod \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") "
Oct 30 22:30:19.893467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893356    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\") pod \"9d579993-8868-4fe9-b437-43fcaa92ddb9\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") "
Oct 30 22:30:19.893467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893451    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\" (UniqueName: \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\") pod \"9d579993-8868-4fe9-b437-43fcaa92ddb9\" (UID: \"9d579993-8868-4fe9-b437-43fcaa92ddb9\") "
Oct 30 22:30:19.893669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893576    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\") pod \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\" (UID: \"1ffe414f-4c3c-45a6-a841-55e4710fefc9\") "
Oct 30 22:30:19.893854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893747    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca" pod "9d579993-8868-4fe9-b437-43fcaa92ddb9" (UID: "9d579993-8868-4fe9-b437-43fcaa92ddb9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.893947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.893926    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca" pod "1ffe414f-4c3c-45a6-a841-55e4710fefc9" (UID: "1ffe414f-4c3c-45a6-a841-55e4710fefc9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:19.894222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.894198    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9d579993-8868-4fe9-b437-43fcaa92ddb9-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.894326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.894226    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1ffe414f-4c3c-45a6-a841-55e4710fefc9-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.895664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.895629    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4" pod "1ffe414f-4c3c-45a6-a841-55e4710fefc9" (UID: "1ffe414f-4c3c-45a6-a841-55e4710fefc9"). InnerVolumeSpecName "kube-api-access-4nmr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.895839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.895818    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l" pod "9d579993-8868-4fe9-b437-43fcaa92ddb9" (UID: "9d579993-8868-4fe9-b437-43fcaa92ddb9"). InnerVolumeSpecName "kube-api-access-s889l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:19.995965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.995899    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nmr4\" (UniqueName: \"kubernetes.io/projected/1ffe414f-4c3c-45a6-a841-55e4710fefc9-kube-api-access-4nmr4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.995965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.995938    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s889l\" (UniqueName: \"kubernetes.io/projected/9d579993-8868-4fe9-b437-43fcaa92ddb9-kube-api-access-s889l\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:19.999250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:19.999220    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:20.651868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.651836    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"
Oct 30 22:30:20.652293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.651857    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb" event={"ID":"1ffe414f-4c3c-45a6-a841-55e4710fefc9","Type":"ContainerDied","Data":"c9c76ffd3ba411f3400ebe89c0085fc2f773ec055914f820dae8ed4aaec6603a"}
Oct 30 22:30:20.652293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.651934    4882 scope.go:122] "RemoveContainer" containerID="87d06b22c97e6b6971f8f94ae4c5cbcf87f5b25a314fc6fe6daa76ba2fd5bf7c"
Oct 30 22:30:20.653557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.653533    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz" event={"ID":"9d579993-8868-4fe9-b437-43fcaa92ddb9","Type":"ContainerDied","Data":"405dace8295ccde21489248db56d4006fb868ae97996c448024b6a9905293c7a"}
Oct 30 22:30:20.653682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.653548    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"
Oct 30 22:30:20.662827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.662798    4882 scope.go:122] "RemoveContainer" containerID="2e857f4db391519d2e8733f8ae708545916b202b1707b5b9834e42d3021ae3d8"
Oct 30 22:30:20.813669 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:20.813629    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:30:20.816461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.816431    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:20.881503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.881465    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:20.887913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.887877    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-praxis-b-7fd427-29889990-d42xb"]
Oct 30 22:30:20.905196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.905105    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\") pod \"73f7c225-adef-4072-a2de-a105c00d2d32\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") "
Oct 30 22:30:20.905375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.905331    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\" (UniqueName: \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\") pod \"73f7c225-adef-4072-a2de-a105c00d2d32\" (UID: \"73f7c225-adef-4072-a2de-a105c00d2d32\") "
Oct 30 22:30:20.905562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.905536    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca" pod "73f7c225-adef-4072-a2de-a105c00d2d32" (UID: "73f7c225-adef-4072-a2de-a105c00d2d32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:20.905710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.905692    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/73f7c225-adef-4072-a2de-a105c00d2d32-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:20.907502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.907477    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc" pod "73f7c225-adef-4072-a2de-a105c00d2d32" (UID: "73f7c225-adef-4072-a2de-a105c00d2d32"). InnerVolumeSpecName "kube-api-access-8vmbc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:20.944523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.944485    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" path="/var/lib/kubelet/pods/1ffe414f-4c3c-45a6-a841-55e4710fefc9/volumes"
Oct 30 22:30:20.945032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.945015    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" path="/var/lib/kubelet/pods/a8edb03c-1bf4-4bac-bbd5-6a7090f57863/volumes"
Oct 30 22:30:20.945390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:20.945378    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" path="/var/lib/kubelet/pods/c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf/volumes"
Oct 30 22:30:21.005937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.005897    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:21.006405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.006383    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8vmbc\" (UniqueName: \"kubernetes.io/projected/73f7c225-adef-4072-a2de-a105c00d2d32-kube-api-access-8vmbc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:21.010048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.010022    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889990-m6mxz"]
Oct 30 22:30:21.017438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.017408    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:30:21.220677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.220569    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:30:21.664750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.664713    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45" event={"ID":"73f7c225-adef-4072-a2de-a105c00d2d32","Type":"ContainerDied","Data":"5b6330514ddad1b6a62269652db7427dc6308157e837e63f38ee67b40d4d385f"}
Oct 30 22:30:21.665243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.664731    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"
Oct 30 22:30:21.665243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.664811    4882 scope.go:122] "RemoveContainer" containerID="b93f594f82bff1f75c4b8fa43423c00e4483d10e58a7ee197f3e64a68e084b4f"
Oct 30 22:30:21.835953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.835910    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:21.840679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:21.840648    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29889990-w8j45"]
Oct 30 22:30:22.943852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:22.943815    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" path="/var/lib/kubelet/pods/73f7c225-adef-4072-a2de-a105c00d2d32/volumes"
Oct 30 22:30:22.944270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:22.944256    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" path="/var/lib/kubelet/pods/9d579993-8868-4fe9-b437-43fcaa92ddb9/volumes"
Oct 30 22:30:26.018244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.018197    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" podUID="66812030-f241-474f-a2ae-49f75d195b20" containerName="payload-processing" containerID="cri-o://d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2" gracePeriod=25
Oct 30 22:30:26.221944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.221883    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" podUID="a3501ece-3569-41f4-811b-b99e61ab5c3e" containerName="payload-pre-processing" containerID="cri-o://3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32" gracePeriod=25
Oct 30 22:30:26.357229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.357201    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:30:26.485793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.485757    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:30:26.491750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.491726    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\") pod \"66812030-f241-474f-a2ae-49f75d195b20\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") "
Oct 30 22:30:26.491829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.491794    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\" (UniqueName: \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\") pod \"66812030-f241-474f-a2ae-49f75d195b20\" (UID: \"66812030-f241-474f-a2ae-49f75d195b20\") "
Oct 30 22:30:26.492241 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.492213    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume" pod "66812030-f241-474f-a2ae-49f75d195b20" (UID: "66812030-f241-474f-a2ae-49f75d195b20"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:26.494076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.494052    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5" pod "66812030-f241-474f-a2ae-49f75d195b20" (UID: "66812030-f241-474f-a2ae-49f75d195b20"). InnerVolumeSpecName "kube-api-access-l8tg5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:26.592572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.592540    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\") pod \"a3501ece-3569-41f4-811b-b99e61ab5c3e\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") "
Oct 30 22:30:26.592776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.592662    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\" (UniqueName: \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\") pod \"a3501ece-3569-41f4-811b-b99e61ab5c3e\" (UID: \"a3501ece-3569-41f4-811b-b99e61ab5c3e\") "
Oct 30 22:30:26.592934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.592910    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume" pod "a3501ece-3569-41f4-811b-b99e61ab5c3e" (UID: "a3501ece-3569-41f4-811b-b99e61ab5c3e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:30:26.593121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.593107    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l8tg5\" (UniqueName: \"kubernetes.io/projected/66812030-f241-474f-a2ae-49f75d195b20-kube-api-access-l8tg5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:26.593171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.593124    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a3501ece-3569-41f4-811b-b99e61ab5c3e-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:26.593171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.593135    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66812030-f241-474f-a2ae-49f75d195b20-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:26.594683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.594666    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js" pod "a3501ece-3569-41f4-811b-b99e61ab5c3e" (UID: "a3501ece-3569-41f4-811b-b99e61ab5c3e"). InnerVolumeSpecName "kube-api-access-qf5js". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:30:26.693891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.693841    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qf5js\" (UniqueName: \"kubernetes.io/projected/a3501ece-3569-41f4-811b-b99e61ab5c3e-kube-api-access-qf5js\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:30:26.708551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.708514    4882 generic.go:356] "Generic (PLEG): container finished" podID="a3501ece-3569-41f4-811b-b99e61ab5c3e" containerID="3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32" exitCode=0
Oct 30 22:30:26.708732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.708558    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"
Oct 30 22:30:26.708732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.708641    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" event={"ID":"a3501ece-3569-41f4-811b-b99e61ab5c3e","Type":"ContainerDied","Data":"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"}
Oct 30 22:30:26.708732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.708718    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx" event={"ID":"a3501ece-3569-41f4-811b-b99e61ab5c3e","Type":"ContainerDied","Data":"47c0476873637adc3de03c18eead9c88dbdefb9763768b6d5f6f2ffb59e022e5"}
Oct 30 22:30:26.708894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.708741    4882 scope.go:122] "RemoveContainer" containerID="3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"
Oct 30 22:30:26.710162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.710141    4882 generic.go:356] "Generic (PLEG): container finished" podID="66812030-f241-474f-a2ae-49f75d195b20" containerID="d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2" exitCode=0
Oct 30 22:30:26.710264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.710186    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" event={"ID":"66812030-f241-474f-a2ae-49f75d195b20","Type":"ContainerDied","Data":"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"}
Oct 30 22:30:26.710264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.710212    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"
Oct 30 22:30:26.710264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.710223    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2" event={"ID":"66812030-f241-474f-a2ae-49f75d195b20","Type":"ContainerDied","Data":"dc1db4964af2f5190d1ee64343a8cb955c98d1f3fd990925212c8cbd409b8792"}
Oct 30 22:30:26.720263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.720227    4882 scope.go:122] "RemoveContainer" containerID="3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"
Oct 30 22:30:26.720561 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:26.720541    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32\": container with ID starting with 3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32 not found: ID does not exist" containerID="3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"
Oct 30 22:30:26.720706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.720570    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32"} err="failed to get container status \"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32\": rpc error: code = NotFound desc = could not find container \"3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32\": container with ID starting with 3fccbb86737c925448f9a4db16431c5526eb0660c1a3a1f07b1b976cdae02f32 not found: ID does not exist"
Oct 30 22:30:26.720706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.720609    4882 scope.go:122] "RemoveContainer" containerID="d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"
Oct 30 22:30:26.730831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.730807    4882 scope.go:122] "RemoveContainer" containerID="d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"
Oct 30 22:30:26.731155 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:26.731134    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2\": container with ID starting with d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2 not found: ID does not exist" containerID="d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"
Oct 30 22:30:26.731235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.731162    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2"} err="failed to get container status \"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2\": rpc error: code = NotFound desc = could not find container \"d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2\": container with ID starting with d45a322c10f524e4023ca2438e912fae2c094521136a1005a991a36f0eae36d2 not found: ID does not exist"
Oct 30 22:30:26.877347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.877267    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:30:26.883852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.883821    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-praxis-b-7fd427-76f9ff478f-74rw2"]
Oct 30 22:30:26.944622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.944561    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66812030-f241-474f-a2ae-49f75d195b20" path="/var/lib/kubelet/pods/66812030-f241-474f-a2ae-49f75d195b20/volumes"
Oct 30 22:30:26.995873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.995837    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:30:26.999991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:26.999962    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-praxis-b-7fd427-6474fb8df5-4xlvx"]
Oct 30 22:30:28.053257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:28.053224    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"]
Oct 30 22:30:28.103441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:28.103404    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-859024a3-jq9ll"]
Oct 30 22:30:28.944182 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:28.944146    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3501ece-3569-41f4-811b-b99e61ab5c3e" path="/var/lib/kubelet/pods/a3501ece-3569-41f4-811b-b99e61ab5c3e/volumes"
Oct 30 22:30:28.944733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:28.944713    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f45c3407-3cbe-4e72-b54b-de04a3446838" path="/var/lib/kubelet/pods/f45c3407-3cbe-4e72-b54b-de04a3446838/volumes"
Oct 30 22:30:38.053221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:38.053142    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"]
Oct 30 22:30:38.103530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:38.103495    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f83b7654-7v8tz"]
Oct 30 22:30:38.943367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:38.943331    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0aef2d8b-0716-4f7a-b906-899853b1abf0" path="/var/lib/kubelet/pods/0aef2d8b-0716-4f7a-b906-899853b1abf0/volumes"
Oct 30 22:30:47.051720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:47.051677    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:30:50.816016 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:30:50.815981    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:30:51.873181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873141    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:30:51.873706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873691    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873707    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873720    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873725    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873732    4882 cpu_manager.go:414] "containerMap: removing container" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873737    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873750    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873755    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.873762 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873764    4882 cpu_manager.go:414] "containerMap: removing container" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873770    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873778    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873782    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873793    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873797    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873803    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873807    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873813    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873817    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873823    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873827    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873835    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a3501ece-3569-41f4-811b-b99e61ab5c3e" containerName="payload-pre-processing"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873841    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3501ece-3569-41f4-811b-b99e61ab5c3e" containerName="payload-pre-processing"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873864    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873869    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873878    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873882    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873889    4882 cpu_manager.go:414] "containerMap: removing container" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873893    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873901    4882 cpu_manager.go:414] "containerMap: removing container" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873905    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873912    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873916    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873926    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873931    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873947    4882 cpu_manager.go:414] "containerMap: removing container" podUID="651f12ba-bf0d-456f-987c-8a91a9caabe1" containerName="revoke-keys"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873952    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="651f12ba-bf0d-456f-987c-8a91a9caabe1" containerName="revoke-keys"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873959    4882 cpu_manager.go:414] "containerMap: removing container" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873964    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873970    4882 cpu_manager.go:414] "containerMap: removing container" podUID="66812030-f241-474f-a2ae-49f75d195b20" containerName="payload-processing"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.873974    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66812030-f241-474f-a2ae-49f75d195b20" containerName="payload-processing"
Oct 30 22:30:51.874039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874052    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874061    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874067    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874072    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874078    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874083    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874088    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874093    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874099    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3501ece-3569-41f4-811b-b99e61ab5c3e" containerName="payload-pre-processing"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874106    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874112    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44a08ce-d8fe-4f8f-8a9e-569f9b76feaf" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874118    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874124    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="73f7c225-adef-4072-a2de-a105c00d2d32" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874130    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874135    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d579993-8868-4fe9-b437-43fcaa92ddb9" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874144    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874150    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="651f12ba-bf0d-456f-987c-8a91a9caabe1" containerName="revoke-keys"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874156    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874160    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="66812030-f241-474f-a2ae-49f75d195b20" containerName="payload-processing"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874166    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.875212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.874171    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.879616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.879595    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:51.937643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.937604    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:30:51.940498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.939237    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-5bb0b1-dockercfg-7nkcm\""
Oct 30 22:30:51.980129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.980097    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-5bb0b1\""
Oct 30 22:30:51.991041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.991003    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghmnb\" (UniqueName: \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:51.991229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.991143    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:51.993839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.993816    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:30:51.994401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994386    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.994450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994402    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.994450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994416    4882 cpu_manager.go:414] "containerMap: removing container" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.994450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994421    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.994450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994430    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.994450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994435    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8edb03c-1bf4-4bac-bbd5-6a7090f57863" containerName="cleanup"
Oct 30 22:30:51.994666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994461    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.994666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994466    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.994666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994546    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ffe414f-4c3c-45a6-a841-55e4710fefc9" containerName="cleanup"
Oct 30 22:30:51.994666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994554    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="954fe549-2577-47f5-b440-e6032657501c" containerName="cleanup"
Oct 30 22:30:51.994666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.994564    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ad13d61-e852-4160-8614-4ac621459382" containerName="cleanup"
Oct 30 22:30:51.998880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:51.998860    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.056428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.056385    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:30:52.095612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.094918    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.095612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.094978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-thfvt\" (UniqueName: \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.095612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.095021    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghmnb\" (UniqueName: \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.095612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.095132    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.096852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.096821    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.166601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.166508    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghmnb\" (UniqueName: \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\") pod \"payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.196402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.196368    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.197964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.196818    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.197964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.196855    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-thfvt\" (UniqueName: \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.197964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.197914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.267341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.267313    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-thfvt\" (UniqueName: \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\") pod \"payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.316101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.315792    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.598241 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:52.598206    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb98d7f63_ed19_42f1_99fe_050591fe84eb.slice/crio-bc0ab01363552bc9bdfc23e3afdba079d6f2cac5958013b7582a14341ed14e3e WatchSource:0}: Error finding container bc0ab01363552bc9bdfc23e3afdba079d6f2cac5958013b7582a14341ed14e3e: Status 404 returned error can't find the container with id bc0ab01363552bc9bdfc23e3afdba079d6f2cac5958013b7582a14341ed14e3e
Oct 30 22:30:52.598692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.598665    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:30:52.706878 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:30:52.706839    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ae872ca_1d79_4b3e_95e0_8c38f0fe0db2.slice/crio-a83940031090d8ac9c48e8a741056de2bf6555b08fa6840e2ff7c08f54cd46c4 WatchSource:0}: Error finding container a83940031090d8ac9c48e8a741056de2bf6555b08fa6840e2ff7c08f54cd46c4: Status 404 returned error can't find the container with id a83940031090d8ac9c48e8a741056de2bf6555b08fa6840e2ff7c08f54cd46c4
Oct 30 22:30:52.710373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.710346    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:30:52.922761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.922647    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" event={"ID":"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2","Type":"ContainerStarted","Data":"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"}
Oct 30 22:30:52.925800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.925770    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:30:52.926006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.925816    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:30:52.926006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.925847    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" event={"ID":"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2","Type":"ContainerStarted","Data":"a83940031090d8ac9c48e8a741056de2bf6555b08fa6840e2ff7c08f54cd46c4"}
Oct 30 22:30:52.926006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.925889    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" event={"ID":"b98d7f63-ed19-42f1-99fe-050591fe84eb","Type":"ContainerStarted","Data":"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"}
Oct 30 22:30:52.926006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:52.925912    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" event={"ID":"b98d7f63-ed19-42f1-99fe-050591fe84eb","Type":"ContainerStarted","Data":"bc0ab01363552bc9bdfc23e3afdba079d6f2cac5958013b7582a14341ed14e3e"}
Oct 30 22:30:53.058045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:53.057988    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" podStartSLOduration=2.057971748 podStartE2EDuration="2.057971748s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:53.053760179 +0000 UTC m=+3176.227084607" watchObservedRunningTime="2026-10-30 22:30:53.057971748 +0000 UTC m=+3176.231296175"
Oct 30 22:30:53.187536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:53.187408    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" podStartSLOduration=2.187381451 podStartE2EDuration="2.187381451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:30:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:30:53.183740449 +0000 UTC m=+3176.357064877" watchObservedRunningTime="2026-10-30 22:30:53.187381451 +0000 UTC m=+3176.360705882"
Oct 30 22:30:53.551877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:53.551792    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"]
Oct 30 22:30:53.555727 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:53.555699    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-praxis-b-7fd427-wf6zv"]
Oct 30 22:30:54.947045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:54.947000    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="651f12ba-bf0d-456f-987c-8a91a9caabe1" path="/var/lib/kubelet/pods/651f12ba-bf0d-456f-987c-8a91a9caabe1/volumes"
Oct 30 22:30:59.257748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:59.257717    4882 scope.go:122] "RemoveContainer" containerID="2730b2b0ea33d87ec938063497edf91ff6a35e2b6d8cdbfa1922570215023577"
Oct 30 22:30:59.268071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:59.268050    4882 scope.go:122] "RemoveContainer" containerID="501dd1f1cbc9857b4a9af96b1a7e8c3eb0d466e38e3857660d3dd635f65c1795"
Oct 30 22:30:59.277793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:59.277771    4882 scope.go:122] "RemoveContainer" containerID="4924fd7181c60a0f7d51a8613a874bad032485a2cc33533329a32f48475fed58"
Oct 30 22:30:59.287439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:59.287416    4882 scope.go:122] "RemoveContainer" containerID="3cfcef73a45e53a11b5d5c148d7c73039b23eecf8a9f2303d4bb162a1dffab23"
Oct 30 22:30:59.297246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:30:59.297225    4882 scope.go:122] "RemoveContainer" containerID="1a282cfcd0980918a51a113875f10570d2260c0f8418044896b2b3fc05062a1f"
Oct 30 22:31:03.418204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.418169    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:31:03.424636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.424574    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.478139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.478098    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:31:03.523547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.523512    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-ae7a1f\""
Oct 30 22:31:03.523998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.523962    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-ae7a1f-dockercfg-wpm6v\""
Oct 30 22:31:03.548434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.548388    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.548671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.548451    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wrbt\" (UniqueName: \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.578184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.578145    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:31:03.582828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.582802    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.623053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.623013    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:31:03.649580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.649546    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.649580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.649608    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wrbt\" (UniqueName: \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.650329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.650306    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.712095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.712019    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wrbt\" (UniqueName: \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\") pod \"payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.738806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.738770    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:03.751117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.751085    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.751260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.751142    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25p6t\" (UniqueName: \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.852713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.852680    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25p6t\" (UniqueName: \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.852928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.852845    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.853528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.853506    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.917782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.917747    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25p6t\" (UniqueName: \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\") pod \"payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:03.937603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:03.937564    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:31:04.053850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.053764    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"]
Oct 30 22:31:04.058049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.058023    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d6c42d6-zbf58"]
Oct 30 22:31:04.198504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.198475    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:04.200989 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:04.200955    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa6c992a_572e_4fb4_a5d0_3bba651d05f7.slice/crio-1740e97538a903511f3d1a200dacee3282cd42fce2ecb0f255b30b9d2b98c704 WatchSource:0}: Error finding container 1740e97538a903511f3d1a200dacee3282cd42fce2ecb0f255b30b9d2b98c704: Status 404 returned error can't find the container with id 1740e97538a903511f3d1a200dacee3282cd42fce2ecb0f255b30b9d2b98c704
Oct 30 22:31:04.204082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.204052    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:31:04.576682 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:04.576651    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89517b8e_a1d9_47c3_a2c3_a2d3f445ed96.slice/crio-86ff5b29e0e9805c7425f64667d656c4bc856f9535704f4bfbaaa1b526be9cf7 WatchSource:0}: Error finding container 86ff5b29e0e9805c7425f64667d656c4bc856f9535704f4bfbaaa1b526be9cf7: Status 404 returned error can't find the container with id 86ff5b29e0e9805c7425f64667d656c4bc856f9535704f4bfbaaa1b526be9cf7
Oct 30 22:31:04.580955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.580931    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:31:04.943180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.943145    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0" path="/var/lib/kubelet/pods/83385fa7-04ef-4945-bbc0-1a0e6c0f7ec0/volumes"
Oct 30 22:31:04.948222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:04.948199    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:31:05.023938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.023893    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" event={"ID":"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96","Type":"ContainerStarted","Data":"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"}
Oct 30 22:31:05.024125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.023967    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" event={"ID":"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96","Type":"ContainerStarted","Data":"86ff5b29e0e9805c7425f64667d656c4bc856f9535704f4bfbaaa1b526be9cf7"}
Oct 30 22:31:05.024125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.023999    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:05.025611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.025577    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" event={"ID":"aa6c992a-572e-4fb4-a5d0-3bba651d05f7","Type":"ContainerStarted","Data":"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"}
Oct 30 22:31:05.025740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.025627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" event={"ID":"aa6c992a-572e-4fb4-a5d0-3bba651d05f7","Type":"ContainerStarted","Data":"1740e97538a903511f3d1a200dacee3282cd42fce2ecb0f255b30b9d2b98c704"}
Oct 30 22:31:05.025740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.025652    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:05.187781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.187720    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" podStartSLOduration=2.187706097 podStartE2EDuration="2.187706097s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:05.186047364 +0000 UTC m=+3188.359371790" watchObservedRunningTime="2026-10-30 22:31:05.187706097 +0000 UTC m=+3188.361030524"
Oct 30 22:31:05.304662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:05.304504    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" podStartSLOduration=2.304484687 podStartE2EDuration="2.304484687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:05.302665422 +0000 UTC m=+3188.475989849" watchObservedRunningTime="2026-10-30 22:31:05.304484687 +0000 UTC m=+3188.477809116"
Oct 30 22:31:16.038499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:16.038470    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:31:16.039049 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:16.038543    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:31:20.449472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.449432    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:31:20.456381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.456355    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.509871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.509826    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:31:20.555454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.555419    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c29159\""
Oct 30 22:31:20.555454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.555419    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c29159-dockercfg-7vtzz\""
Oct 30 22:31:20.570604 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.570552    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:31:20.575490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.575457    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.579915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.579877    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccdw6\" (UniqueName: \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.580095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.579938    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.642880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.642841    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:31:20.681207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.681173    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bzc6w\" (UniqueName: \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.681439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.681218    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.681439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.681336    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccdw6\" (UniqueName: \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.681439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.681390    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.682119 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.682099    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.749946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.749866    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccdw6\" (UniqueName: \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\") pod \"payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.783523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.783475    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bzc6w\" (UniqueName: \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.783712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.783553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.783806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.783704    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:20.784291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.784267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.818193 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:31:20.818154    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:31:20.846728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.846694    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bzc6w\" (UniqueName: \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\") pod \"payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:20.892997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:20.892364    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:21.172664 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:21.172612    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0810636_154a_461e_a74f_46e0e456b172.slice/crio-91d47c94b4043ded6942f765c13546b54e3f192d3078144cb75a0b09c93969f3 WatchSource:0}: Error finding container 91d47c94b4043ded6942f765c13546b54e3f192d3078144cb75a0b09c93969f3: Status 404 returned error can't find the container with id 91d47c94b4043ded6942f765c13546b54e3f192d3078144cb75a0b09c93969f3
Oct 30 22:31:21.178394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:21.178366    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:31:21.325052 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:21.325016    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0958b46_bb85_422d_8b90_8212df761c7d.slice/crio-467e63b6ed999eb657c79173ef7f8ab01e4d38754e2bbbd44fe8d52850969dc7 WatchSource:0}: Error finding container 467e63b6ed999eb657c79173ef7f8ab01e4d38754e2bbbd44fe8d52850969dc7: Status 404 returned error can't find the container with id 467e63b6ed999eb657c79173ef7f8ab01e4d38754e2bbbd44fe8d52850969dc7
Oct 30 22:31:21.331639 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:21.328869    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:31:22.156315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.156282    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" event={"ID":"f0810636-154a-461e-a74f-46e0e456b172","Type":"ContainerStarted","Data":"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"}
Oct 30 22:31:22.156864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.156332    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" event={"ID":"f0810636-154a-461e-a74f-46e0e456b172","Type":"ContainerStarted","Data":"91d47c94b4043ded6942f765c13546b54e3f192d3078144cb75a0b09c93969f3"}
Oct 30 22:31:22.156864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.156739    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:22.158337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.158311    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" event={"ID":"f0958b46-bb85-422d-8b90-8212df761c7d","Type":"ContainerStarted","Data":"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"}
Oct 30 22:31:22.158483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.158353    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" event={"ID":"f0958b46-bb85-422d-8b90-8212df761c7d","Type":"ContainerStarted","Data":"467e63b6ed999eb657c79173ef7f8ab01e4d38754e2bbbd44fe8d52850969dc7"}
Oct 30 22:31:22.158483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.158392    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:22.294358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.294303    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" podStartSLOduration=2.2942890240000002 podStartE2EDuration="2.294289024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:22.289550501 +0000 UTC m=+3205.462874928" watchObservedRunningTime="2026-10-30 22:31:22.294289024 +0000 UTC m=+3205.467613451"
Oct 30 22:31:22.416580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:22.416461    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" podStartSLOduration=2.416439831 podStartE2EDuration="2.416439831s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:22.411946476 +0000 UTC m=+3205.585270904" watchObservedRunningTime="2026-10-30 22:31:22.416439831 +0000 UTC m=+3205.589764260"
Oct 30 22:31:23.069404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:23.069356    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"]
Oct 30 22:31:23.073307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:23.073279    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7d6988d1-nbfqh"]
Oct 30 22:31:24.943902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:24.943866    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abd6cdee-2341-43bd-a4b8-7c3d78e95081" path="/var/lib/kubelet/pods/abd6cdee-2341-43bd-a4b8-7c3d78e95081/volumes"
Oct 30 22:31:27.419120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.419079    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:31:27.424003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.423976    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.478524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.478487    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:31:27.523753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.523708    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-355213-dockercfg-dzsj7\""
Oct 30 22:31:27.523753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.523706    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-355213\""
Oct 30 22:31:27.564821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.564784    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p544k\" (UniqueName: \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.565004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.564973    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.593754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.593721    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:31:27.597953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.597929    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.673106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.673012    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.673303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.673147    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p544k\" (UniqueName: \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.674766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.674736    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.676316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.676286    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:31:27.757239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.757198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p544k\" (UniqueName: \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\") pod \"payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") " pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:27.774553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.774521    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lrcv\" (UniqueName: \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.774712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.774697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.875428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.875381    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lrcv\" (UniqueName: \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.875686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.875536    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.876388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.876364    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:27.937203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:27.937134    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lrcv\" (UniqueName: \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\") pod \"payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:28.041759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:28.041727    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:28.211806 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:28.211717    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:28.416314 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:28.416286    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00ddcc84_9394_4cb9_a963_2fa624021fd2.slice/crio-5b9ac2597c3a1786c8f0462df44e02b2cd6e4d31514263a93167bf95db7a38d5 WatchSource:0}: Error finding container 5b9ac2597c3a1786c8f0462df44e02b2cd6e4d31514263a93167bf95db7a38d5: Status 404 returned error can't find the container with id 5b9ac2597c3a1786c8f0462df44e02b2cd6e4d31514263a93167bf95db7a38d5
Oct 30 22:31:28.420383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:28.420349    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:31:28.592540 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:28.592504    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1fe04a_04cd_4435_8ba8_2692b5dba62c.slice/crio-9d242b2f2789f97fb4a866766c62d74ff592850d97587968499c21b4f63ca478 WatchSource:0}: Error finding container 9d242b2f2789f97fb4a866766c62d74ff592850d97587968499c21b4f63ca478: Status 404 returned error can't find the container with id 9d242b2f2789f97fb4a866766c62d74ff592850d97587968499c21b4f63ca478
Oct 30 22:31:28.597676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:28.597649    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:31:29.214758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.214718    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" event={"ID":"bc1fe04a-04cd-4435-8ba8-2692b5dba62c","Type":"ContainerStarted","Data":"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"}
Oct 30 22:31:29.214934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.214778    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" event={"ID":"bc1fe04a-04cd-4435-8ba8-2692b5dba62c","Type":"ContainerStarted","Data":"9d242b2f2789f97fb4a866766c62d74ff592850d97587968499c21b4f63ca478"}
Oct 30 22:31:29.214934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.214808    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:29.216732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.216697    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" event={"ID":"00ddcc84-9394-4cb9-a963-2fa624021fd2","Type":"ContainerStarted","Data":"a4d3f5169d329d7eba7950dbab165f0afc34b7ca3425b1e6d41d1ab464d76228"}
Oct 30 22:31:29.216900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.216745    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" event={"ID":"00ddcc84-9394-4cb9-a963-2fa624021fd2","Type":"ContainerStarted","Data":"5b9ac2597c3a1786c8f0462df44e02b2cd6e4d31514263a93167bf95db7a38d5"}
Oct 30 22:31:29.216900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.216765    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:29.334795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.334743    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" podStartSLOduration=2.334730273 podStartE2EDuration="2.334730273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:29.33422921 +0000 UTC m=+3212.507553649" watchObservedRunningTime="2026-10-30 22:31:29.334730273 +0000 UTC m=+3212.508054718"
Oct 30 22:31:29.460377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:29.460320    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" podStartSLOduration=2.460304037 podStartE2EDuration="2.460304037s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:29.459220714 +0000 UTC m=+3212.632545141" watchObservedRunningTime="2026-10-30 22:31:29.460304037 +0000 UTC m=+3212.633628462"
Oct 30 22:31:33.053086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:33.053050    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"]
Oct 30 22:31:33.072480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:33.072439    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0d905bb9-ch4m7"]
Oct 30 22:31:33.168184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:33.168155    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:31:33.168375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:33.168295    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:31:34.944854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:34.944809    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a2e3d8f-dffb-4a99-b302-da2af97beb4c" path="/var/lib/kubelet/pods/1a2e3d8f-dffb-4a99-b302-da2af97beb4c/volumes"
Oct 30 22:31:35.257075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:35.256981    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:31:38.361696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.361657    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:31:38.367154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.367127    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.421324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.421290    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:31:38.442180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.442144    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w1-323082\""
Oct 30 22:31:38.465810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.465778    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w1-323082-dockercfg-xvd75\""
Oct 30 22:31:38.491181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.491150    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:31:38.495548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.495526    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.502085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.502056    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.502332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.502305    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srvl\" (UniqueName: \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.594710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.594664    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:31:38.604218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.604163    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srvl\" (UniqueName: \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.604446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.604314    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.604446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.604395    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.604612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.604446    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2xfh\" (UniqueName: \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.605780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.605752    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.666943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.666862    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srvl\" (UniqueName: \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\") pod \"payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.682581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.682540    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:38.705937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.705907    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.706102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.705962    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2xfh\" (UniqueName: \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.706853 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.706823    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.775256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.775216    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2xfh\" (UniqueName: \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\") pod \"payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:38.809518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:38.809487    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:39.073908 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:39.073875    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c8f639f_422a_431e_b36a_61e49aa4c618.slice/crio-5bc8757aed6d82dcd7a7dca717d7cbeb12289041e454e45d2ec34ea5e3dd6a63 WatchSource:0}: Error finding container 5bc8757aed6d82dcd7a7dca717d7cbeb12289041e454e45d2ec34ea5e3dd6a63: Status 404 returned error can't find the container with id 5bc8757aed6d82dcd7a7dca717d7cbeb12289041e454e45d2ec34ea5e3dd6a63
Oct 30 22:31:39.079157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.079130    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"]
Oct 30 22:31:39.083399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.083372    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:31:39.087047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.087017    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fa89ed4e-skmvf"]
Oct 30 22:31:39.194464 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:31:39.194431    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794ee7b6_6c2e_4821_8edf_e3c3f8b1cab6.slice/crio-d0bf4c6c442417fba5386b8a5f52cc87522b99d14bba48d6310f7487f638fda9 WatchSource:0}: Error finding container d0bf4c6c442417fba5386b8a5f52cc87522b99d14bba48d6310f7487f638fda9: Status 404 returned error can't find the container with id d0bf4c6c442417fba5386b8a5f52cc87522b99d14bba48d6310f7487f638fda9
Oct 30 22:31:39.198151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.198125    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:31:39.294914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.294874    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" event={"ID":"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6","Type":"ContainerStarted","Data":"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"}
Oct 30 22:31:39.295090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.294946    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:39.295090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.294970    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" event={"ID":"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6","Type":"ContainerStarted","Data":"d0bf4c6c442417fba5386b8a5f52cc87522b99d14bba48d6310f7487f638fda9"}
Oct 30 22:31:39.296781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.296752    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" event={"ID":"9c8f639f-422a-431e-b36a-61e49aa4c618","Type":"ContainerStarted","Data":"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"}
Oct 30 22:31:39.296931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.296801    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" event={"ID":"9c8f639f-422a-431e-b36a-61e49aa4c618","Type":"ContainerStarted","Data":"5bc8757aed6d82dcd7a7dca717d7cbeb12289041e454e45d2ec34ea5e3dd6a63"}
Oct 30 22:31:39.296931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.296830    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:39.418960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.418908    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" podStartSLOduration=1.418893341 podStartE2EDuration="1.418893341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:39.415011633 +0000 UTC m=+3222.588336059" watchObservedRunningTime="2026-10-30 22:31:39.418893341 +0000 UTC m=+3222.592218069"
Oct 30 22:31:39.544067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:39.543969    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" podStartSLOduration=1.543940452 podStartE2EDuration="1.543940452s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:31:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:31:39.538819795 +0000 UTC m=+3222.712144225" watchObservedRunningTime="2026-10-30 22:31:39.543940452 +0000 UTC m=+3222.717264918"
Oct 30 22:31:40.227022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:40.226988    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:31:40.227237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:40.227059    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:31:40.943907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:40.943872    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae8f4de-e64a-49b2-9ee2-c6e87aa06365" path="/var/lib/kubelet/pods/0ae8f4de-e64a-49b2-9ee2-c6e87aa06365/volumes"
Oct 30 22:31:46.054522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:46.054484    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"]
Oct 30 22:31:46.104682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:46.104639    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b18d84-qs7k9"]
Oct 30 22:31:46.944515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:46.944477    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="479ca62c-0315-4dbb-a6ae-27a4cea765fa" path="/var/lib/kubelet/pods/479ca62c-0315-4dbb-a6ae-27a4cea765fa/volumes"
Oct 30 22:31:50.052975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:50.052943    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"]
Oct 30 22:31:50.103650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:50.103614    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-421b1b1d-6j58v"]
Oct 30 22:31:50.308032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:50.307936    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:31:50.308032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:50.308021    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:31:50.808011 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:31:50.807966    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:31:50.946524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:50.946476    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10319863-96b0-42d7-b95e-c2cff963dc4c" path="/var/lib/kubelet/pods/10319863-96b0-42d7-b95e-c2cff963dc4c/volumes"
Oct 30 22:31:59.065117 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.065080    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"]
Oct 30 22:31:59.116074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.116027    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8af63b7f-csxq6"]
Oct 30 22:31:59.398848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.398804    4882 scope.go:122] "RemoveContainer" containerID="92deee345eeeed5cd736d2a0e87bb2c112d7bcc0182cbe6426315bca242bdaf4"
Oct 30 22:31:59.409470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.409449    4882 scope.go:122] "RemoveContainer" containerID="72ccbf4275cce1a9ba41ccf05f440dbe062c3794f7d5415f9cef8f500e4dcc7a"
Oct 30 22:31:59.419323 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.419301    4882 scope.go:122] "RemoveContainer" containerID="dd475929aa8678ad4426c8db17b02119ae1c9e3c6e10f61dffa3336cec8715bb"
Oct 30 22:31:59.428874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.428847    4882 scope.go:122] "RemoveContainer" containerID="d9bf1e864d2630c92c8487065cbecd8ff99a9d78ee63a1ceae0790d427ebbf0b"
Oct 30 22:31:59.438447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:31:59.438426    4882 scope.go:122] "RemoveContainer" containerID="6d812f67dc3657c36b7174b4266e0d2174906d332cad481ce65e9a9c80bd3330"
Oct 30 22:32:00.944089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:00.944053    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f5402867-a37f-49a3-8a7f-03a8e5ed4824" path="/var/lib/kubelet/pods/f5402867-a37f-49a3-8a7f-03a8e5ed4824/volumes"
Oct 30 22:32:09.055034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:09.054949    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"]
Oct 30 22:32:09.060478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:09.060443    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7116d5f4-q6pdf"]
Oct 30 22:32:10.945057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:10.945017    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0633118b-ec01-48d0-bc10-d3ba30db86d8" path="/var/lib/kubelet/pods/0633118b-ec01-48d0-bc10-d3ba30db86d8/volumes"
Oct 30 22:32:19.601286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:19.601251    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:32:20.809281 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:32:20.809238    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:32:22.055697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:22.055661    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"]
Oct 30 22:32:22.105729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:22.105696    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10a9200d-lqzb7"]
Oct 30 22:32:22.945338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:22.945291    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="648b24a4-8d62-48af-aa49-477f145f03c2" path="/var/lib/kubelet/pods/648b24a4-8d62-48af-aa49-477f145f03c2/volumes"
Oct 30 22:32:46.059259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:46.059213    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"]
Oct 30 22:32:46.102511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:46.102472    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0765b5c1-59kqd"]
Oct 30 22:32:46.944449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:46.944404    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3560054-1d06-4c2d-ae7b-ad2a29540ea0" path="/var/lib/kubelet/pods/f3560054-1d06-4c2d-ae7b-ad2a29540ea0/volumes"
Oct 30 22:32:50.554004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.553960    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:32:50.558864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.558841    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.600984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.600945    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601041    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601126    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601231    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601265    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601376    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601437    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbdm\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601474    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.601692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.601532    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.612171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.612140    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-355213-openshift-default-dockercfg-k56hc\""
Oct 30 22:32:50.668879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.668822    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:32:50.702483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702435    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.702483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702486    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702524    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702555    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbdm\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702621    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702658    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702735    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702765    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.702829    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.703005    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.703052    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.703082    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.703207    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.703578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.703558    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.704988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.704969    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.705243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.705221    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.764786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.764750    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.765031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.764830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbdm\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\") pod \"e2e-explicit-355213-openshift-default-6f7c585497-62cld\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") " pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:50.806711 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:32:50.806558    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:32:50.873542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:50.873506    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:51.303542 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:32:51.303510    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1aa1e08_8443_4342_bd1c_1dd8bc60dcdd.slice/crio-1c0679c79878b86ff63e4899b111aae38e9b4a5abc310325a5075341ed26a7a8 WatchSource:0}: Error finding container 1c0679c79878b86ff63e4899b111aae38e9b4a5abc310325a5075341ed26a7a8: Status 404 returned error can't find the container with id 1c0679c79878b86ff63e4899b111aae38e9b4a5abc310325a5075341ed26a7a8
Oct 30 22:32:51.306501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.306465    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:32:51.306634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.306537    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:32:51.306634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.306569    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:32:51.307218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.307192    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:32:51.864107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.864064    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" event={"ID":"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd","Type":"ContainerStarted","Data":"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"}
Oct 30 22:32:51.864566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.864134    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" event={"ID":"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd","Type":"ContainerStarted","Data":"1c0679c79878b86ff63e4899b111aae38e9b4a5abc310325a5075341ed26a7a8"}
Oct 30 22:32:51.874174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.874146    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:51.988941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:51.988880    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podStartSLOduration=1.9888642220000001 podStartE2EDuration="1.988864222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:32:51.985212849 +0000 UTC m=+3295.158537277" watchObservedRunningTime="2026-10-30 22:32:51.988864222 +0000 UTC m=+3295.162188649"
Oct 30 22:32:52.875812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:52.875780    4882 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-62cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 22:32:52.876216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:52.875829    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 22:32:53.874697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:53.874657    4882 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-62cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 22:32:53.874897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:53.874711    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 22:32:54.874802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:54.874763    4882 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-62cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 22:32:54.875205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:54.874834    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 22:32:55.874408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:55.874372    4882 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-62cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 22:32:55.874644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:55.874455    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 22:32:56.874148 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:56.874112    4882 patch_prober.go:29] interesting pod/e2e-explicit-355213-openshift-default-6f7c585497-62cld container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused" start-of-body=
Oct 30 22:32:56.874616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:56.874169    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.24:15021/healthz/ready\": dial tcp 10.132.1.24:15021: connect: connection refused"
Oct 30 22:32:57.420828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:57.420795    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:32:57.461217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:57.461191    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:32:57.953798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:57.953748    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:57.954348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:57.954209    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:57.955152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:57.955134    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:32:59.055628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.055578    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"]
Oct 30 22:32:59.061327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.061300    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6d81b914-nvh5d"]
Oct 30 22:32:59.300865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.300829    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:32:59.305153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.305132    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.377628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.375406    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-5e884c\"/\"auth-test-model-5e884c-kserve-self-signed-certs\""
Oct 30 22:32:59.377628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.375476    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-5e884c\"/\"openshift-service-ca.crt\""
Oct 30 22:32:59.377628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.375689    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w2-5e884c\"/\"kube-root-ca.crt\""
Oct 30 22:32:59.377628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.376523    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:32:59.382275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.382226    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w2-5e884c\"/\"default-dockercfg-6d7bs\""
Oct 30 22:32:59.440379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440340    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.440572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440497    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.440572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440562    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.440693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440662    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.440786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440767    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fmt4\" (UniqueName: \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.440824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.440811    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.489170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.489140    4882 scope.go:122] "RemoveContainer" containerID="0079089cbd1053437e244b28f1a7172593c2490739a43d2c73910335ac53ea3e"
Oct 30 22:32:59.500081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.500054    4882 scope.go:122] "RemoveContainer" containerID="82cc80ecf6ab4f3c53a18a26ab2c26d848290964945a25b2afec063d52d3871f"
Oct 30 22:32:59.511050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.511029    4882 scope.go:122] "RemoveContainer" containerID="0e76ee1995c8e3304f940f61aa25094c763f267cdc9969ad85cfb9a37e617de3"
Oct 30 22:32:59.521988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.521961    4882 scope.go:122] "RemoveContainer" containerID="69bede1820be38a0def9ee36066a0d933ae406ff5fd4adede18a1e64054ad23e"
Oct 30 22:32:59.535168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.535132    4882 scope.go:122] "RemoveContainer" containerID="701ebb59eae2dac4377ec1eb86109c76708d7c462ce16dd68e1358a691603192"
Oct 30 22:32:59.542211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542182    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542285    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fmt4\" (UniqueName: \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542325    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542390    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542500    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542688    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.542882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.542853    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.543263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.543239    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.544914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.544887    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.545286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.545252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.545509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.545490    4882 scope.go:122] "RemoveContainer" containerID="bc531d4ac2b989532ad2e7c5c54baa4c8189cb22bd6affa2b57959e80541378b"
Oct 30 22:32:59.571997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.571975    4882 scope.go:122] "RemoveContainer" containerID="ddd1c61cfe92858008881587f496c82b82abc10354020ae1a89a34ca9548e133"
Oct 30 22:32:59.659110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.659068    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fmt4\" (UniqueName: \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\") pod \"auth-test-model-5e884c-kserve-f8587685-8rkh4\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") " pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:32:59.918438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:32:59.918395    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:33:00.303891 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:33:00.303854    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod61d7d6fb_2bfd_406e_85d6_d5e2d2f0a3da.slice/crio-9362bd13bc1385909d7f88072d4f721548677610a8f726291a1057b3913b7e1e WatchSource:0}: Error finding container 9362bd13bc1385909d7f88072d4f721548677610a8f726291a1057b3913b7e1e: Status 404 returned error can't find the container with id 9362bd13bc1385909d7f88072d4f721548677610a8f726291a1057b3913b7e1e
Oct 30 22:33:00.307245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:00.307219    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:33:00.944221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:00.944187    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f" path="/var/lib/kubelet/pods/6fe0a43c-6a9d-4fc5-9ce3-80be390d8b4f/volumes"
Oct 30 22:33:00.944581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:00.944562    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" event={"ID":"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da","Type":"ContainerStarted","Data":"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"}
Oct 30 22:33:00.944648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:00.944604    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" event={"ID":"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da","Type":"ContainerStarted","Data":"9362bd13bc1385909d7f88072d4f721548677610a8f726291a1057b3913b7e1e"}
Oct 30 22:33:01.052081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:01.052048    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"]
Oct 30 22:33:01.057399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:01.057371    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-732fc0fd-6jj6w"]
Oct 30 22:33:01.098886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:01.098830    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" podStartSLOduration=2.098815809 podStartE2EDuration="2.098815809s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:32:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:33:01.058720816 +0000 UTC m=+3304.232045243" watchObservedRunningTime="2026-10-30 22:33:01.098815809 +0000 UTC m=+3304.272140285"
Oct 30 22:33:01.919556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:01.919519    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:33:02.944046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:02.944012    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398fc6aa-9510-461a-a45d-2898e6f42118" path="/var/lib/kubelet/pods/398fc6aa-9510-461a-a45d-2898e6f42118/volumes"
Oct 30 22:33:09.919567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:09.919530    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:33:09.933157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:09.933128    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:33:10.019030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:10.018996    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:33:11.052339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:11.052308    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"]
Oct 30 22:33:11.102248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:11.102213    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5977c5c3-nzt57"]
Oct 30 22:33:12.943366 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:12.943334    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f28780d4-c968-4a31-a718-48fdc7e5dd3b" path="/var/lib/kubelet/pods/f28780d4-c968-4a31-a718-48fdc7e5dd3b/volumes"
Oct 30 22:33:14.731230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:14.731182    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:33:20.809931 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:33:20.809889    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:33:22.062442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:22.062408    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"]
Oct 30 22:33:22.111298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:22.111259    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-511ea286-dvwgb"]
Oct 30 22:33:22.943327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:22.943287    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="149523c7-a00a-4a3f-90bc-9a90f9cb0695" path="/var/lib/kubelet/pods/149523c7-a00a-4a3f-90bc-9a90f9cb0695/volumes"
Oct 30 22:33:23.051099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:23.051056    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"]
Oct 30 22:33:23.054619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:23.054573    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6946e2f-4mpdh"]
Oct 30 22:33:24.944214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:24.944181    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="218e8468-e91e-42d5-82ab-7af4eaae0afc" path="/var/lib/kubelet/pods/218e8468-e91e-42d5-82ab-7af4eaae0afc/volumes"
Oct 30 22:33:26.051056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:26.051023    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"]
Oct 30 22:33:26.101158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:26.101122    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a72476f6-ljsrz"]
Oct 30 22:33:26.943783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:26.943744    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c50176-9c1f-450f-af80-7e4c60015e37" path="/var/lib/kubelet/pods/28c50176-9c1f-450f-af80-7e4c60015e37/volumes"
Oct 30 22:33:27.050260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:27.050220    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"]
Oct 30 22:33:27.100379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:27.100341    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e7a0be76-8ntc9"]
Oct 30 22:33:28.943388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:28.943352    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="95ef4854-825e-4247-a406-ae6ecc12d109" path="/var/lib/kubelet/pods/95ef4854-825e-4247-a406-ae6ecc12d109/volumes"
Oct 30 22:33:36.052344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:36.052308    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"]
Oct 30 22:33:36.102501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:36.102466    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-44f13885-sdtxh"]
Oct 30 22:33:36.944731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:36.944652    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d636af08-7e3d-41bd-b0da-3d8160149812" path="/var/lib/kubelet/pods/d636af08-7e3d-41bd-b0da-3d8160149812/volumes"
Oct 30 22:33:41.053194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:41.053157    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"]
Oct 30 22:33:41.103701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:41.103665    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a17cce00-9cvkt"]
Oct 30 22:33:42.943843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:42.943809    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cab449cb-c32f-4452-8ede-00eaf2f35a34" path="/var/lib/kubelet/pods/cab449cb-c32f-4452-8ede-00eaf2f35a34/volumes"
Oct 30 22:33:47.051295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:47.051257    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"]
Oct 30 22:33:47.101907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:47.101870    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d260d2a5-bkdhn"]
Oct 30 22:33:48.051507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:48.051471    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"]
Oct 30 22:33:48.101941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:48.101907    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cc261a01-clqrr"]
Oct 30 22:33:48.943992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:48.943958    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7d722b80-0dba-41eb-a237-cf79931643d1" path="/var/lib/kubelet/pods/7d722b80-0dba-41eb-a237-cf79931643d1/volumes"
Oct 30 22:33:48.944497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:48.944472    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b17afd6-1430-455e-a927-05bd45497c3b" path="/var/lib/kubelet/pods/8b17afd6-1430-455e-a927-05bd45497c3b/volumes"
Oct 30 22:33:50.809259 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:33:50.809222    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:33:59.632267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.632232    4882 scope.go:122] "RemoveContainer" containerID="c10be09f42d581ce7c5ca3adc4b523691d492dabd430757bbbcbbb98bf0fb3a9"
Oct 30 22:33:59.642682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.642653    4882 scope.go:122] "RemoveContainer" containerID="70458bfa268a4b734e6aed1105f7b189844948aaea6db48e293c3c6763f90f00"
Oct 30 22:33:59.651766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.651743    4882 scope.go:122] "RemoveContainer" containerID="0849ff8459b95e0a3e305e4ed37e9516f59e8d968a2913fdfcdc0a7fbbac0742"
Oct 30 22:33:59.661713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.661688    4882 scope.go:122] "RemoveContainer" containerID="cb3ee9b78e4380b99a7f4c3c55dfd23f21477dce22e0d701f711fed97244d827"
Oct 30 22:33:59.671218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.671193    4882 scope.go:122] "RemoveContainer" containerID="2c2478522bb13719e56fbb43148114993c39e0a4fb68654b9a02ef53fc202067"
Oct 30 22:33:59.680473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.680451    4882 scope.go:122] "RemoveContainer" containerID="3071c6d0a62377bdee2c16da23bc27c1b3b107100a0e79e4d68e44b2a7614e3d"
Oct 30 22:33:59.690149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:33:59.690128    4882 scope.go:122] "RemoveContainer" containerID="ef72278ca8406220253020335010232a782cd87a7590b898c6328f3d8b467897"
Oct 30 22:34:06.301767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.301731    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:34:06.310452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.310413    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.365124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.365081    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:34:06.401918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.401862    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.402099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.401926    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-78pd6\" (UniqueName: \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.402157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.402093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.402157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.402124    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.402291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.402204    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.402291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.402249    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.410306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.410279    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-43d592\"/\"auth-test-model-43d592-kserve-self-signed-certs\""
Oct 30 22:34:06.410461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.410331    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-43d592\"/\"kube-root-ca.crt\""
Oct 30 22:34:06.410461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.410356    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-43d592\"/\"default-dockercfg-m99cr\""
Oct 30 22:34:06.414053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.414032    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-43d592\"/\"openshift-service-ca.crt\""
Oct 30 22:34:06.504122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504082    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504125    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504164    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504181    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504251    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504270    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-78pd6\" (UniqueName: \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504519    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504626    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.504763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.504704    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.506651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.506626    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.506856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.506836    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.575440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.575359    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-78pd6\" (UniqueName: \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\") pod \"auth-test-model-43d592-kserve-7f76f46cc7-v5mqw\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") " pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:06.624448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:06.624408    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:07.010230 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:34:07.010192    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod977372c6_4596_4c33_bedd_98ab46e2aba8.slice/crio-5fe499fc9954f43de338126c88b157f6c4d60756d92b19400ec24cb7fec15a05 WatchSource:0}: Error finding container 5fe499fc9954f43de338126c88b157f6c4d60756d92b19400ec24cb7fec15a05: Status 404 returned error can't find the container with id 5fe499fc9954f43de338126c88b157f6c4d60756d92b19400ec24cb7fec15a05
Oct 30 22:34:07.013114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:07.013094    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:34:07.424087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:07.424047    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" event={"ID":"977372c6-4596-4c33-bedd-98ab46e2aba8","Type":"ContainerStarted","Data":"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"}
Oct 30 22:34:07.424545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:07.424099    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" event={"ID":"977372c6-4596-4c33-bedd-98ab46e2aba8","Type":"ContainerStarted","Data":"5fe499fc9954f43de338126c88b157f6c4d60756d92b19400ec24cb7fec15a05"}
Oct 30 22:34:07.551925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:07.551866    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" podStartSLOduration=1.55184626 podStartE2EDuration="1.55184626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:07.548161511 +0000 UTC m=+3370.721485937" watchObservedRunningTime="2026-10-30 22:34:07.55184626 +0000 UTC m=+3370.725170685"
Oct 30 22:34:07.625208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:07.625169    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:16.624656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:16.624615    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:16.637737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:16.637708    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:17.508875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:17.508841    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:34:17.670446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:17.670411    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:34:19.535309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.535275    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:34:19.539760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.539736    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589232    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589276    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpgc\" (UniqueName: \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589311    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589514    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589543    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.589696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.589672    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.593196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.593175    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6fc60c\"/\"openshift-service-ca.crt\""
Oct 30 22:34:19.643637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.643578    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6fc60c\"/\"auth-test-model-6fc60c-kserve-self-signed-certs\""
Oct 30 22:34:19.643637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.643575    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w0-6fc60c\"/\"default-dockercfg-52ws7\""
Oct 30 22:34:19.643866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.643618    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w0-6fc60c\"/\"kube-root-ca.crt\""
Oct 30 22:34:19.647275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.647249    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:34:19.691314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691280    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.691527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691323    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpgc\" (UniqueName: \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.691527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691364    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.691527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691435    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.691527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691468    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.692576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691540    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.692576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691803    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.692576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691961    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.692576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.691975    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.694132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.694106    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.694236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.694221    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.754283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.754244    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpgc\" (UniqueName: \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\") pod \"auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") " pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:19.853447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:19.853401    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:20.240655 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:34:20.240488    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda425850_6851_4880_a556_8a7a2bf61bc1.slice/crio-dee96a006df83d79153fe35f3b87061ad5a180ee83c62822eb1115f45bbcbf3f WatchSource:0}: Error finding container dee96a006df83d79153fe35f3b87061ad5a180ee83c62822eb1115f45bbcbf3f: Status 404 returned error can't find the container with id dee96a006df83d79153fe35f3b87061ad5a180ee83c62822eb1115f45bbcbf3f
Oct 30 22:34:20.243336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:20.243308    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:34:20.515932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:20.515838    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" event={"ID":"da425850-6851-4880-a556-8a7a2bf61bc1","Type":"ContainerStarted","Data":"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"}
Oct 30 22:34:20.515932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:20.515884    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" event={"ID":"da425850-6851-4880-a556-8a7a2bf61bc1","Type":"ContainerStarted","Data":"dee96a006df83d79153fe35f3b87061ad5a180ee83c62822eb1115f45bbcbf3f"}
Oct 30 22:34:20.638288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:20.638235    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" podStartSLOduration=1.638218293 podStartE2EDuration="1.638218293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:20.634775375 +0000 UTC m=+3383.808099802" watchObservedRunningTime="2026-10-30 22:34:20.638218293 +0000 UTC m=+3383.811542721"
Oct 30 22:34:20.817736 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:34:20.817652    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:34:20.853809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:20.853763    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:22.639507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:22.639471    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:34:22.640101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:22.639697    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy" containerID="cri-o://d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b" gracePeriod=30
Oct 30 22:34:27.935947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.935916    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:34:27.993408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993371    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993444    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993482    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993627 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993530    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993700    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:27.993767 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993736    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993799    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:27.993897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993883    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.993955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993938    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.994017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993886    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:27.994017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.993975    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.994017 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994006    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\") pod \"8fb98270-b727-419a-a50c-ebce91f90863\" (UID: \"8fb98270-b727-419a-a50c-ebce91f90863\") "
Oct 30 22:34:27.994251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994226    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:34:27.994251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994234    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:27.994654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994629    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8fb98270-b727-419a-a50c-ebce91f90863-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:27.994766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994658    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:27.994766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994675    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:27.994766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994689    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:27.994766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.994706    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:27.996239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.996214    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:34:27.996381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.996349    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:27.996491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.996400    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:34:27.996491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:27.996432    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h" pod "8fb98270-b727-419a-a50c-ebce91f90863" (UID: "8fb98270-b727-419a-a50c-ebce91f90863"). InnerVolumeSpecName "kube-api-access-9sh4h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:34:28.096450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.096413    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8fb98270-b727-419a-a50c-ebce91f90863-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.096450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.096446    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.096450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.096456    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8fb98270-b727-419a-a50c-ebce91f90863-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.096826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.096468    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9sh4h\" (UniqueName: \"kubernetes.io/projected/8fb98270-b727-419a-a50c-ebce91f90863-kube-api-access-9sh4h\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:34:28.576316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.576281    4882 generic.go:356] "Generic (PLEG): container finished" podID="8fb98270-b727-419a-a50c-ebce91f90863" containerID="d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b" exitCode=0
Oct 30 22:34:28.576715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.576326    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" event={"ID":"8fb98270-b727-419a-a50c-ebce91f90863","Type":"ContainerDied","Data":"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"}
Oct 30 22:34:28.576715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.576329    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"
Oct 30 22:34:28.576715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.576368    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv" event={"ID":"8fb98270-b727-419a-a50c-ebce91f90863","Type":"ContainerDied","Data":"93db18008ba9487f0017d4dcdbe35c491792751df0648ac8e8d60990eb768d6b"}
Oct 30 22:34:28.576715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.576392    4882 scope.go:122] "RemoveContainer" containerID="d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"
Oct 30 22:34:28.590862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.590839    4882 scope.go:122] "RemoveContainer" containerID="d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"
Oct 30 22:34:28.591211 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:34:28.591188    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b\": container with ID starting with d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b not found: ID does not exist" containerID="d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"
Oct 30 22:34:28.591287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.591219    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b"} err="failed to get container status \"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b\": rpc error: code = NotFound desc = could not find container \"d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b\": container with ID starting with d84f35ee602ac5e6ec7d50998b731e8e2ffadd0c8385f4589457af53346f0e1b not found: ID does not exist"
Oct 30 22:34:28.746473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.746436    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:34:28.796690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.796656    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-56f3fe-openshift-default-784cd69b87-xtgnv"]
Oct 30 22:34:28.944306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:28.944274    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8fb98270-b727-419a-a50c-ebce91f90863" path="/var/lib/kubelet/pods/8fb98270-b727-419a-a50c-ebce91f90863/volumes"
Oct 30 22:34:29.853842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:29.853805    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:29.866346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:29.866315    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:30.603254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:30.603224    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:34:50.810406 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:34:50.810368    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:34:55.454570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:55.454534    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:34:57.750875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.750830    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:34:57.751791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.751773    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy"
Oct 30 22:34:57.751845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.751794    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy"
Oct 30 22:34:57.752000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.751989    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8fb98270-b727-419a-a50c-ebce91f90863" containerName="istio-proxy"
Oct 30 22:34:57.757089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.757067    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.808239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.808199    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:34:57.854800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.854772    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-355213\"/\"explicit-model-355213-kserve-self-signed-certs\""
Oct 30 22:34:57.854984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.854837    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-355213\"/\"default-dockercfg-f7s2z\""
Oct 30 22:34:57.860294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.860273    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-355213\"/\"openshift-service-ca.crt\""
Oct 30 22:34:57.860348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.860273    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-355213\"/\"kube-root-ca.crt\""
Oct 30 22:34:57.872028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.871992    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.872172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.872107    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnjkx\" (UniqueName: \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.872172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.872145    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.872309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.872289    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.872356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.872325    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.872471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.872452    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973387    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973519    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnjkx\" (UniqueName: \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973556    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973707    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973744    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973868    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.973955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.973883    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.974110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.974087    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.974181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.974165    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.976124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.976099    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:57.976246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:57.976161    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:58.036196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.036104    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnjkx\" (UniqueName: \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\") pod \"explicit-model-355213-kserve-54f65b6477-gvcqv\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") " pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:58.071178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.071141    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:58.454580 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:34:58.454543    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e43950e_c523_4b59_8fab_45371d2970c6.slice/crio-483464887d8ef918502179bd0bd59d4de50d0f732c7544a9a46cdb7be3cbfca1 WatchSource:0}: Error finding container 483464887d8ef918502179bd0bd59d4de50d0f732c7544a9a46cdb7be3cbfca1: Status 404 returned error can't find the container with id 483464887d8ef918502179bd0bd59d4de50d0f732c7544a9a46cdb7be3cbfca1
Oct 30 22:34:58.457175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.457148    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:34:58.780365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.780273    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" event={"ID":"1e43950e-c523-4b59-8fab-45371d2970c6","Type":"ContainerStarted","Data":"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"}
Oct 30 22:34:58.780365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.780327    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" event={"ID":"1e43950e-c523-4b59-8fab-45371d2970c6","Type":"ContainerStarted","Data":"483464887d8ef918502179bd0bd59d4de50d0f732c7544a9a46cdb7be3cbfca1"}
Oct 30 22:34:58.904765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:58.904710    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" podStartSLOduration=1.90469121 podStartE2EDuration="1.90469121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:34:58.90410113 +0000 UTC m=+3422.077425548" watchObservedRunningTime="2026-10-30 22:34:58.90469121 +0000 UTC m=+3422.078015638"
Oct 30 22:34:59.071360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.071271    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:34:59.751849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.751808    4882 scope.go:122] "RemoveContainer" containerID="91e39e66fa687ecd54318ee8ad6d21355e4a2aca10f0abe4e1312973edb0881c"
Oct 30 22:34:59.762213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.761973    4882 scope.go:122] "RemoveContainer" containerID="03965f536f2e29a315f82aa2cbda1f540c8f3ddf17fef383291c412b5645cba0"
Oct 30 22:34:59.772118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.772095    4882 scope.go:122] "RemoveContainer" containerID="ef4c63df06eabffd04859833dd4ea5d7c2e530d08b88df6003731d881281403f"
Oct 30 22:34:59.782388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.782370    4882 scope.go:122] "RemoveContainer" containerID="c5bf7641294cfd41a745063df9ebdb65481d869c7f25ebaac6bc3548e77db592"
Oct 30 22:34:59.793130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.793110    4882 scope.go:122] "RemoveContainer" containerID="24613bc186f80f260d23de3e060e280e819a834962d8c788d31c7f096bb06912"
Oct 30 22:34:59.803264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.803238    4882 scope.go:122] "RemoveContainer" containerID="648de49670db1016de7a61e81dc700dfe9720977e00184ba3635a575fec45a57"
Oct 30 22:34:59.813235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.813212    4882 scope.go:122] "RemoveContainer" containerID="b8f0983f9948b2b007ec7e0af9bac86718936305485214a44684a2b3e99e68a4"
Oct 30 22:34:59.823547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.823527    4882 scope.go:122] "RemoveContainer" containerID="cb1ff8bb7ec3ac972e38be387309678de71b36fbf34ffd9e80b70995dbc9457f"
Oct 30 22:34:59.832955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:34:59.832933    4882 scope.go:122] "RemoveContainer" containerID="e3fbe0a0a1522661dae12ab652a2dfa18f317764c7098b80e314c37fe6e8c7dd"
Oct 30 22:35:08.072144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:08.072045    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:35:08.085557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:08.085530    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:35:08.865429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:08.865397    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:35:13.254046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:13.254008    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:35:13.254645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:13.254247    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy" containerID="cri-o://76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec" gracePeriod=30
Oct 30 22:35:18.537061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.536987    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:35:18.638768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.638730    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.638955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.638808    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.638955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.638860    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.638955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.638879    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.638955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.638922    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.639137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639025    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.639137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.639253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639144    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.639253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639182    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\") pod \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\" (UID: \"372d3d7c-84d3-4a4d-9088-e1c1061ad657\") "
Oct 30 22:35:18.639253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639195    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:18.639253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639228    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:18.639495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639324    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:18.639495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639373    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:18.639674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639619    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:18.639912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639887    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.640014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639917    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.640014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639933    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.640014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639949    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.640014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.639964    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.641319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.641288    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:18.641428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.641368    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:35:18.641428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.641405    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "kube-api-access-f58j6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:18.641507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.641472    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy" pod "372d3d7c-84d3-4a4d-9088-e1c1061ad657" (UID: "372d3d7c-84d3-4a4d-9088-e1c1061ad657"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:18.742090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.742047    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.742090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.742092    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.742315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.742108    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58j6\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-kube-api-access-f58j6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.742315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.742122    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/372d3d7c-84d3-4a4d-9088-e1c1061ad657-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:18.924950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.924910    4882 generic.go:356] "Generic (PLEG): container finished" podID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerID="76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec" exitCode=0
Oct 30 22:35:18.925178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.924969    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"
Oct 30 22:35:18.925178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.925008    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" event={"ID":"372d3d7c-84d3-4a4d-9088-e1c1061ad657","Type":"ContainerDied","Data":"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"}
Oct 30 22:35:18.925178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.925074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf" event={"ID":"372d3d7c-84d3-4a4d-9088-e1c1061ad657","Type":"ContainerDied","Data":"18da2b969466e41a4cc7b097ac26fe70a3456a38cc149e79dd344d7e561eb295"}
Oct 30 22:35:18.925178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.925107    4882 scope.go:122] "RemoveContainer" containerID="76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"
Oct 30 22:35:18.940799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.940004    4882 scope.go:122] "RemoveContainer" containerID="76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"
Oct 30 22:35:18.940799 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:18.940388    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec\": container with ID starting with 76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec not found: ID does not exist" containerID="76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"
Oct 30 22:35:18.940799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:18.940431    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec"} err="failed to get container status \"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec\": rpc error: code = NotFound desc = could not find container \"76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec\": container with ID starting with 76c340993f6f2c12a7c78c19a5fc83fa772d82445eb8fddea09480ccef2a86ec not found: ID does not exist"
Oct 30 22:35:19.095992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:19.095954    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:35:19.099172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:19.099141    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-l4xqf"]
Oct 30 22:35:20.809955 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:20.809914    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:35:20.943481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:20.943446    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" path="/var/lib/kubelet/pods/372d3d7c-84d3-4a4d-9088-e1c1061ad657/volumes"
Oct 30 22:35:25.302313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.302278    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:35:25.302909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.302894    4882 cpu_manager.go:414] "containerMap: removing container" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy"
Oct 30 22:35:25.302956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.302910    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy"
Oct 30 22:35:25.303030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.303021    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="372d3d7c-84d3-4a4d-9088-e1c1061ad657" containerName="istio-proxy"
Oct 30 22:35:25.308143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.308120    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.362307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.362272    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:35:25.408181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.408150    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5bb0b1\"/\"openshift-service-ca.crt\""
Oct 30 22:35:25.408181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.408170    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w2-5bb0b1\"/\"kube-root-ca.crt\""
Oct 30 22:35:25.411342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.411319    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5bb0b1\"/\"auth-test-model-5bb0b1-kserve-self-signed-certs\""
Oct 30 22:35:25.411342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.411322    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w2-5bb0b1\"/\"default-dockercfg-2p482\""
Oct 30 22:35:25.429070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429044    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.429191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429121    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.429259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429237    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xswdv\" (UniqueName: \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.429362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429344    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.429426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429415    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.429482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.429468    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.530557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530521    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.530797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530619    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.530797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530660    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xswdv\" (UniqueName: \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.530797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530734    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.530797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530769    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.531034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.530816    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.531213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.531183    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.531361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.531216    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.531435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.531374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.532926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.532906    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.533565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.533544    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.592696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.592661    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xswdv\" (UniqueName: \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\") pod \"auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") " pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:25.622135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:25.622099    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:26.009982 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:35:26.009945    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod28a5c2a9_49b3_4122_8a4a_bae9c3b8c113.slice/crio-f94437605b504d8aae26ade18c270f55891571b78f786870f1f810952932eeda WatchSource:0}: Error finding container f94437605b504d8aae26ade18c270f55891571b78f786870f1f810952932eeda: Status 404 returned error can't find the container with id f94437605b504d8aae26ade18c270f55891571b78f786870f1f810952932eeda
Oct 30 22:35:26.012883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:26.012858    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:35:26.980348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:26.980314    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" event={"ID":"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113","Type":"ContainerStarted","Data":"b73a99a7ba3363e4f3597a27703e7be768f3875de225058be5c791d5d8d2bb9f"}
Oct 30 22:35:26.980902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:26.980354    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" event={"ID":"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113","Type":"ContainerStarted","Data":"f94437605b504d8aae26ade18c270f55891571b78f786870f1f810952932eeda"}
Oct 30 22:35:27.099147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:27.099060    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" podStartSLOduration=2.099043879 podStartE2EDuration="2.099043879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:27.098838492 +0000 UTC m=+3450.272162919" watchObservedRunningTime="2026-10-30 22:35:27.099043879 +0000 UTC m=+3450.272368308"
Oct 30 22:35:27.622573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:27.622533    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:34.563549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:34.563506    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:35:35.622598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:35.622547    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:35.636023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:35.635995    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:36.054954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.054875    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:35:36.437295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.437262    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"]
Oct 30 22:35:36.441459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.441437    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.494415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.494376    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:35:36.494647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.494618    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:35:36.496389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.496360    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"]
Oct 30 22:35:36.569327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.569292    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.569493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.569407    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.569548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.569493    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6tph\" (UniqueName: \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.671143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.671091    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.671572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.671223    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.671572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.671321    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6tph\" (UniqueName: \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.672025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.672003    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.673670 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.673640    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.734500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.734423    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6tph\" (UniqueName: \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\") pod \"maas-api-revoke-keys-e2e-explicit-355213-6qvqd\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:36.756884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:36.756849    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:37.142789 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:35:37.142755    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod794d79d8_92c3_4358_a276_79f251a21db7.slice/crio-fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e WatchSource:0}: Error finding container fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e: Status 404 returned error can't find the container with id fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e
Oct 30 22:35:37.156693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:37.156661    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"]
Oct 30 22:35:38.057844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:38.057797    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd" event={"ID":"794d79d8-92c3-4358-a276-79f251a21db7","Type":"ContainerStarted","Data":"3de6d1ebb041b5abcb85bb71569e0115e3a544558da6f98173c921e98c30b6bd"}
Oct 30 22:35:38.058308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:38.057862    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd" event={"ID":"794d79d8-92c3-4358-a276-79f251a21db7","Type":"ContainerStarted","Data":"fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e"}
Oct 30 22:35:38.175528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:38.175460    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd" podStartSLOduration=2.175440738 podStartE2EDuration="2.175440738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:35:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:35:38.174488755 +0000 UTC m=+3461.347813173" watchObservedRunningTime="2026-10-30 22:35:38.175440738 +0000 UTC m=+3461.348765167"
Oct 30 22:35:39.066056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:39.066019    4882 generic.go:356] "Generic (PLEG): container finished" podID="794d79d8-92c3-4358-a276-79f251a21db7" containerID="3de6d1ebb041b5abcb85bb71569e0115e3a544558da6f98173c921e98c30b6bd" exitCode=0
Oct 30 22:35:39.066551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:39.066104    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd" event={"ID":"794d79d8-92c3-4358-a276-79f251a21db7","Type":"ContainerDied","Data":"3de6d1ebb041b5abcb85bb71569e0115e3a544558da6f98173c921e98c30b6bd"}
Oct 30 22:35:40.237339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.237313    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:40.320066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.320027    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\") pod \"794d79d8-92c3-4358-a276-79f251a21db7\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") "
Oct 30 22:35:40.320265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.320114    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\") pod \"794d79d8-92c3-4358-a276-79f251a21db7\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") "
Oct 30 22:35:40.320265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.320187    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\" (UniqueName: \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\") pod \"794d79d8-92c3-4358-a276-79f251a21db7\" (UID: \"794d79d8-92c3-4358-a276-79f251a21db7\") "
Oct 30 22:35:40.320526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.320500    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca" pod "794d79d8-92c3-4358-a276-79f251a21db7" (UID: "794d79d8-92c3-4358-a276-79f251a21db7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:40.320788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.320759    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:40.322314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.322285    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca" pod "794d79d8-92c3-4358-a276-79f251a21db7" (UID: "794d79d8-92c3-4358-a276-79f251a21db7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:40.322414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.322369    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph" pod "794d79d8-92c3-4358-a276-79f251a21db7" (UID: "794d79d8-92c3-4358-a276-79f251a21db7"). InnerVolumeSpecName "kube-api-access-x6tph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:40.421546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.421508    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/794d79d8-92c3-4358-a276-79f251a21db7-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:40.421546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:40.421540    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6tph\" (UniqueName: \"kubernetes.io/projected/794d79d8-92c3-4358-a276-79f251a21db7-kube-api-access-x6tph\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:41.080129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:41.080099    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"
Oct 30 22:35:41.080315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:41.080102    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd" event={"ID":"794d79d8-92c3-4358-a276-79f251a21db7","Type":"ContainerDied","Data":"fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e"}
Oct 30 22:35:41.080315 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:41.080217    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb4d4366d7022fdeb6a097b5dd3e3e818a90c9ea4cbe0c14bdb31bf2b1a9d54e"
Oct 30 22:35:41.869646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:41.869611    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:35:47.949384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:47.949350    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:35:48.062079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:48.060819    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:35:50.808787 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:50.808746    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:35:52.952498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:52.952431    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" podUID="00ddcc84-9394-4cb9-a963-2fa624021fd2" containerName="payload-processing" containerID="cri-o://a4d3f5169d329d7eba7950dbab165f0afc34b7ca3425b1e6d41d1ab464d76228" gracePeriod=25
Oct 30 22:35:53.061607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.061533    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" podUID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" containerName="payload-pre-processing" containerID="cri-o://c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22" gracePeriod=25
Oct 30 22:35:53.163458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.163416    4882 generic.go:356] "Generic (PLEG): container finished" podID="00ddcc84-9394-4cb9-a963-2fa624021fd2" containerID="a4d3f5169d329d7eba7950dbab165f0afc34b7ca3425b1e6d41d1ab464d76228" exitCode=0
Oct 30 22:35:53.163642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.163491    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" event={"ID":"00ddcc84-9394-4cb9-a963-2fa624021fd2","Type":"ContainerDied","Data":"a4d3f5169d329d7eba7950dbab165f0afc34b7ca3425b1e6d41d1ab464d76228"}
Oct 30 22:35:53.240824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.240789    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:35:53.282520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.282478    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\" (UniqueName: \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\") pod \"00ddcc84-9394-4cb9-a963-2fa624021fd2\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") "
Oct 30 22:35:53.282735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.282572    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\") pod \"00ddcc84-9394-4cb9-a963-2fa624021fd2\" (UID: \"00ddcc84-9394-4cb9-a963-2fa624021fd2\") "
Oct 30 22:35:53.283074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.282952    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume" pod "00ddcc84-9394-4cb9-a963-2fa624021fd2" (UID: "00ddcc84-9394-4cb9-a963-2fa624021fd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:53.283463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.283260    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00ddcc84-9394-4cb9-a963-2fa624021fd2-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:53.284909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.284876    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k" pod "00ddcc84-9394-4cb9-a963-2fa624021fd2" (UID: "00ddcc84-9394-4cb9-a963-2fa624021fd2"). InnerVolumeSpecName "kube-api-access-p544k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:53.329371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.329342    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:35:53.384657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.384613    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\") pod \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") "
Oct 30 22:35:53.384868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.384713    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\" (UniqueName: \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\") pod \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\" (UID: \"bc1fe04a-04cd-4435-8ba8-2692b5dba62c\") "
Oct 30 22:35:53.385067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.385038    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume" pod "bc1fe04a-04cd-4435-8ba8-2692b5dba62c" (UID: "bc1fe04a-04cd-4435-8ba8-2692b5dba62c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:35:53.385207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.385182    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:53.385207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.385210    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p544k\" (UniqueName: \"kubernetes.io/projected/00ddcc84-9394-4cb9-a963-2fa624021fd2-kube-api-access-p544k\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:53.386960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.386928    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv" pod "bc1fe04a-04cd-4435-8ba8-2692b5dba62c" (UID: "bc1fe04a-04cd-4435-8ba8-2692b5dba62c"). InnerVolumeSpecName "kube-api-access-4lrcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:53.487358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:53.487284    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lrcv\" (UniqueName: \"kubernetes.io/projected/bc1fe04a-04cd-4435-8ba8-2692b5dba62c-kube-api-access-4lrcv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:54.172052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.172012    4882 generic.go:356] "Generic (PLEG): container finished" podID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" containerID="c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22" exitCode=0
Oct 30 22:35:54.172535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.172063    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"
Oct 30 22:35:54.172535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.172109    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" event={"ID":"bc1fe04a-04cd-4435-8ba8-2692b5dba62c","Type":"ContainerDied","Data":"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"}
Oct 30 22:35:54.172535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.172175    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz" event={"ID":"bc1fe04a-04cd-4435-8ba8-2692b5dba62c","Type":"ContainerDied","Data":"9d242b2f2789f97fb4a866766c62d74ff592850d97587968499c21b4f63ca478"}
Oct 30 22:35:54.172535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.172214    4882 scope.go:122] "RemoveContainer" containerID="c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"
Oct 30 22:35:54.173795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.173779    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"
Oct 30 22:35:54.173795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.173786    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc" event={"ID":"00ddcc84-9394-4cb9-a963-2fa624021fd2","Type":"ContainerDied","Data":"5b9ac2597c3a1786c8f0462df44e02b2cd6e4d31514263a93167bf95db7a38d5"}
Oct 30 22:35:54.185307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.185289    4882 scope.go:122] "RemoveContainer" containerID="c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"
Oct 30 22:35:54.185636 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:54.185613    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22\": container with ID starting with c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22 not found: ID does not exist" containerID="c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"
Oct 30 22:35:54.185690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.185647    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22"} err="failed to get container status \"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22\": rpc error: code = NotFound desc = could not find container \"c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22\": container with ID starting with c0798a1e0c7b8a437ce4ae5ddec260025d534f55264c64fa51e9786512092a22 not found: ID does not exist"
Oct 30 22:35:54.185690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.185666    4882 scope.go:122] "RemoveContainer" containerID="a4d3f5169d329d7eba7950dbab165f0afc34b7ca3425b1e6d41d1ab464d76228"
Oct 30 22:35:54.343783 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.343748    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:35:54.349792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.349767    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-355213-6bdf467d8c-6wmnc"]
Oct 30 22:35:54.504663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.504623    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:35:54.508302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.508276    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-355213-7457cdd947-pxhhz"]
Oct 30 22:35:54.943782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.943744    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00ddcc84-9394-4cb9-a963-2fa624021fd2" path="/var/lib/kubelet/pods/00ddcc84-9394-4cb9-a963-2fa624021fd2/volumes"
Oct 30 22:35:54.944221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:54.944207    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" path="/var/lib/kubelet/pods/bc1fe04a-04cd-4435-8ba8-2692b5dba62c/volumes"
Oct 30 22:35:56.895096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:56.895054    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" podUID="1e43950e-c523-4b59-8fab-45371d2970c6" containerName="main" containerID="cri-o://289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2" gracePeriod=45
Oct 30 22:35:57.007914 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:57.007857    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e43950e_c523_4b59_8fab_45371d2970c6.slice/crio-289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1e43950e_c523_4b59_8fab_45371d2970c6.slice/crio-conmon-289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:35:57.191958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.191926    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:35:57.195357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.195324    4882 generic.go:356] "Generic (PLEG): container finished" podID="1e43950e-c523-4b59-8fab-45371d2970c6" containerID="289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2" exitCode=0
Oct 30 22:35:57.195500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.195394    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"
Oct 30 22:35:57.195500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.195408    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" event={"ID":"1e43950e-c523-4b59-8fab-45371d2970c6","Type":"ContainerDied","Data":"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"}
Oct 30 22:35:57.195500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.195449    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv" event={"ID":"1e43950e-c523-4b59-8fab-45371d2970c6","Type":"ContainerDied","Data":"483464887d8ef918502179bd0bd59d4de50d0f732c7544a9a46cdb7be3cbfca1"}
Oct 30 22:35:57.195500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.195466    4882 scope.go:122] "RemoveContainer" containerID="289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"
Oct 30 22:35:57.205220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.205198    4882 scope.go:122] "RemoveContainer" containerID="289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"
Oct 30 22:35:57.205487 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:35:57.205467    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2\": container with ID starting with 289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2 not found: ID does not exist" containerID="289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"
Oct 30 22:35:57.205535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.205495    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2"} err="failed to get container status \"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2\": rpc error: code = NotFound desc = could not find container \"289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2\": container with ID starting with 289405d57cd7b5b164c8e3c026256e07f22ad333933e9414d767e92517499bf2 not found: ID does not exist"
Oct 30 22:35:57.234635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.234601    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.234815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.234692    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.234815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.234743    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.234815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.234795    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.234995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.234871    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\" (UniqueName: \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.235057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235007    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\") pod \"1e43950e-c523-4b59-8fab-45371d2970c6\" (UID: \"1e43950e-c523-4b59-8fab-45371d2970c6\") "
Oct 30 22:35:57.235259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235229    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:57.235350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235244    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:57.235402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235343    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:57.235700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235682    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.235768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235704    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.235768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.235716    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.237445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.237413    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "kube-api-access-wnjkx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:35:57.237549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.237461    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:35:57.237549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.237481    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs" pod "1e43950e-c523-4b59-8fab-45371d2970c6" (UID: "1e43950e-c523-4b59-8fab-45371d2970c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:35:57.336988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.336955    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnjkx\" (UniqueName: \"kubernetes.io/projected/1e43950e-c523-4b59-8fab-45371d2970c6-kube-api-access-wnjkx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.336988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.336987    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1e43950e-c523-4b59-8fab-45371d2970c6-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.337196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.336997    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1e43950e-c523-4b59-8fab-45371d2970c6-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:35:57.618236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.618199    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:35:57.622208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:57.622174    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-355213/explicit-model-355213-kserve-54f65b6477-gvcqv"]
Oct 30 22:35:58.685282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:58.685249    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"]
Oct 30 22:35:58.739299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:58.739259    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-355213-6qvqd"]
Oct 30 22:35:58.951533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:58.951434    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e43950e-c523-4b59-8fab-45371d2970c6" path="/var/lib/kubelet/pods/1e43950e-c523-4b59-8fab-45371d2970c6/volumes"
Oct 30 22:35:58.952177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:35:58.952155    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794d79d8-92c3-4358-a276-79f251a21db7" path="/var/lib/kubelet/pods/794d79d8-92c3-4358-a276-79f251a21db7/volumes"
Oct 30 22:36:06.291258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:06.291210    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:36:06.291808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:06.291545    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy" containerID="cri-o://45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73" gracePeriod=30
Oct 30 22:36:11.576184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.576119    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:36:11.720287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720247    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720305    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720381    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720421    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720458    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720550    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:11.720789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720549    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:11.720789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720677    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.720789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720739    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.721047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720818    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.721047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720830    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:11.721047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.720895    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\") pod \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\" (UID: \"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd\") "
Oct 30 22:36:11.721292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721095    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:11.721398 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721370    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:11.721668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721641    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.721803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721671    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.721803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721689    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.721803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721702    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.721803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.721717    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.723543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.723518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:11.723698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.723614    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:11.723698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.723662    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:11.723825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.723771    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm" pod "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" (UID: "c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd"). InnerVolumeSpecName "kube-api-access-pcbdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:11.823205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.823164    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.823205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.823196    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.823205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.823210    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbdm\" (UniqueName: \"kubernetes.io/projected/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-kube-api-access-pcbdm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:11.823467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:11.823223    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:12.295329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.295291    4882 generic.go:356] "Generic (PLEG): container finished" podID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerID="45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73" exitCode=0
Oct 30 22:36:12.295522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.295363    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"
Oct 30 22:36:12.295522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.295375    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" event={"ID":"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd","Type":"ContainerDied","Data":"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"}
Oct 30 22:36:12.295522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.295442    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld" event={"ID":"c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd","Type":"ContainerDied","Data":"1c0679c79878b86ff63e4899b111aae38e9b4a5abc310325a5075341ed26a7a8"}
Oct 30 22:36:12.295522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.295472    4882 scope.go:122] "RemoveContainer" containerID="45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"
Oct 30 22:36:12.306445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.306426    4882 scope.go:122] "RemoveContainer" containerID="45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"
Oct 30 22:36:12.306810 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:12.306784    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73\": container with ID starting with 45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73 not found: ID does not exist" containerID="45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"
Oct 30 22:36:12.306880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.306816    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73"} err="failed to get container status \"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73\": rpc error: code = NotFound desc = could not find container \"45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73\": container with ID starting with 45e352d5e7232c5f4f74589698ec3b81d219d3e0b1b370ad8ab43128e1269c73 not found: ID does not exist"
Oct 30 22:36:12.464081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.464042    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:36:12.514156 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.514117    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-355213-openshift-default-6f7c585497-62cld"]
Oct 30 22:36:12.944004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:12.943961    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" path="/var/lib/kubelet/pods/c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd/volumes"
Oct 30 22:36:20.808895 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:20.808854    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:36:33.003834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:33.003793    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:36:38.640689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:38.640598    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:36:38.641137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:38.640806    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy" containerID="cri-o://1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260" gracePeriod=30
Oct 30 22:36:43.927503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:43.927425    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:36:43.935165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:43.935138    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:36:43.935393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:43.935368    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy" containerID="cri-o://db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2" gracePeriod=30
Oct 30 22:36:44.020045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020000    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020205 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020121    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020237    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020273    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020303    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020337    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020390    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020454    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.020574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020514    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\") pod \"f95518bf-37c2-446d-a964-aaa56b988b0b\" (UID: \"f95518bf-37c2-446d-a964-aaa56b988b0b\") "
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020805    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.020911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021035    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021215    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021287    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021311    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.021350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021327    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f95518bf-37c2-446d-a964-aaa56b988b0b-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.021967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.021937    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:44.022784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.022758    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:44.023268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.023242    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "kube-api-access-nf6qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:44.023360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.023332    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:44.024036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.024014    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token" pod "f95518bf-37c2-446d-a964-aaa56b988b0b" (UID: "f95518bf-37c2-446d-a964-aaa56b988b0b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:44.122368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122330    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.122368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122365    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nf6qr\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-kube-api-access-nf6qr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.122368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122377    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.122681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122385    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.122681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122395    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.122681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.122403    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f95518bf-37c2-446d-a964-aaa56b988b0b-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:44.514819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.514783    4882 generic.go:356] "Generic (PLEG): container finished" podID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerID="1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260" exitCode=0
Oct 30 22:36:44.515087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.514832    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"
Oct 30 22:36:44.515087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.514865    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" event={"ID":"f95518bf-37c2-446d-a964-aaa56b988b0b","Type":"ContainerDied","Data":"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"}
Oct 30 22:36:44.515087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.514922    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt" event={"ID":"f95518bf-37c2-446d-a964-aaa56b988b0b","Type":"ContainerDied","Data":"3213eb2ec58146f288c8cba905ec3a5de39758fea162ace95558f66aea4900fd"}
Oct 30 22:36:44.515087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.514944    4882 scope.go:122] "RemoveContainer" containerID="1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"
Oct 30 22:36:44.525952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.525933    4882 scope.go:122] "RemoveContainer" containerID="1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"
Oct 30 22:36:44.526343 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:44.526312    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260\": container with ID starting with 1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260 not found: ID does not exist" containerID="1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"
Oct 30 22:36:44.526407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.526357    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260"} err="failed to get container status \"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260\": rpc error: code = NotFound desc = could not find container \"1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260\": container with ID starting with 1d14cffc6b78e6dcda544d26ef8c00e48b66abc76fd5b8c554ff8f3d9a518260 not found: ID does not exist"
Oct 30 22:36:44.683601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.683554    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:36:44.733744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.733708    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-5zjwt"]
Oct 30 22:36:44.800232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.800147    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:36:44.800408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.800364    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy" containerID="cri-o://bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067" gracePeriod=30
Oct 30 22:36:44.944107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:44.944069    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" path="/var/lib/kubelet/pods/f95518bf-37c2-446d-a964-aaa56b988b0b/volumes"
Oct 30 22:36:49.208526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.208460    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:36:49.281581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281548    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281600    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281701    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281775    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281855    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281885    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281888    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.281982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281922    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.281982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.281979    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.282201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282024    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\") pod \"dc329956-0457-467a-a679-7ee12ae8fcd8\" (UID: \"dc329956-0457-467a-a679-7ee12ae8fcd8\") "
Oct 30 22:36:49.282201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282177    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.282319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282293    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:49.282487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282395    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.282659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282502    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.282961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282937    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.283044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282968    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc329956-0457-467a-a679-7ee12ae8fcd8-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.283044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282985    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.283044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.282999    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.283044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.283014    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.284137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.284112    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:49.284270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.284247    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "kube-api-access-jn2ss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:49.284435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.284416    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:49.284435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.284422    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy" pod "dc329956-0457-467a-a679-7ee12ae8fcd8" (UID: "dc329956-0457-467a-a679-7ee12ae8fcd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:49.384084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.384051    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.384084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.384082    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.384084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.384092    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc329956-0457-467a-a679-7ee12ae8fcd8-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.384336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.384102    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn2ss\" (UniqueName: \"kubernetes.io/projected/dc329956-0457-467a-a679-7ee12ae8fcd8-kube-api-access-jn2ss\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:49.550120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.550031    4882 generic.go:356] "Generic (PLEG): container finished" podID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerID="db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2" exitCode=0
Oct 30 22:36:49.550120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.550095    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"
Oct 30 22:36:49.550365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.550112    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" event={"ID":"dc329956-0457-467a-a679-7ee12ae8fcd8","Type":"ContainerDied","Data":"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"}
Oct 30 22:36:49.550365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.550166    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf" event={"ID":"dc329956-0457-467a-a679-7ee12ae8fcd8","Type":"ContainerDied","Data":"5048fa92a5528b669f72d25ba2fa874a611a772b41147a1de22130869cf396b2"}
Oct 30 22:36:49.550365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.550190    4882 scope.go:122] "RemoveContainer" containerID="db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"
Oct 30 22:36:49.561503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.561484    4882 scope.go:122] "RemoveContainer" containerID="db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"
Oct 30 22:36:49.561841 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:49.561813    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2\": container with ID starting with db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2 not found: ID does not exist" containerID="db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"
Oct 30 22:36:49.561897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.561853    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2"} err="failed to get container status \"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2\": rpc error: code = NotFound desc = could not find container \"db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2\": container with ID starting with db0b5ffa335771e861715804ecc77a186954fb601bce93131bc29636d2d0f8a2 not found: ID does not exist"
Oct 30 22:36:49.728461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.728428    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:36:49.732195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:49.732167    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-t9xlf"]
Oct 30 22:36:50.074334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.074267    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:36:50.195248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195212    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195307    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195367    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195410    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195518    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195550    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195621    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195696    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:36:50.195862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195703    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.195862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.195777    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\") pod \"7ab871ca-51ca-4f5f-8069-82c09968709e\" (UID: \"7ab871ca-51ca-4f5f-8069-82c09968709e\") "
Oct 30 22:36:50.196376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.196353    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7ab871ca-51ca-4f5f-8069-82c09968709e-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.197539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.197469    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:50.198048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.197982    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:50.198797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.198575    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:36:50.198797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.198615    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:50.199034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.199005    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:50.199371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.199351    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:50.199534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.199517    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:36:50.201321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.201290    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8" pod "7ab871ca-51ca-4f5f-8069-82c09968709e" (UID: "7ab871ca-51ca-4f5f-8069-82c09968709e"). InnerVolumeSpecName "kube-api-access-hvxw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:36:50.297151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297109    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvxw8\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-kube-api-access-hvxw8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297146    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297158    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297167    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297178    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297187    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7ab871ca-51ca-4f5f-8069-82c09968709e-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297197    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.297676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.297205    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7ab871ca-51ca-4f5f-8069-82c09968709e-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:36:50.558255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.558224    4882 generic.go:356] "Generic (PLEG): container finished" podID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerID="bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067" exitCode=0
Oct 30 22:36:50.558468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.558273    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"
Oct 30 22:36:50.558468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.558312    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" event={"ID":"7ab871ca-51ca-4f5f-8069-82c09968709e","Type":"ContainerDied","Data":"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"}
Oct 30 22:36:50.558468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.558370    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4" event={"ID":"7ab871ca-51ca-4f5f-8069-82c09968709e","Type":"ContainerDied","Data":"001643ede1d1ade5299c25eff356773cbead2f64a1d9629a8048af4d3040879a"}
Oct 30 22:36:50.558468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.558404    4882 scope.go:122] "RemoveContainer" containerID="bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"
Oct 30 22:36:50.569188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.569168    4882 scope.go:122] "RemoveContainer" containerID="bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"
Oct 30 22:36:50.569523 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:50.569504    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067\": container with ID starting with bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067 not found: ID does not exist" containerID="bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"
Oct 30 22:36:50.569575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.569531    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067"} err="failed to get container status \"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067\": rpc error: code = NotFound desc = could not find container \"bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067\": container with ID starting with bffe68ac139df03ebcaf01b73f129bac6a7307527f0697948296c2b89e1eb067 not found: ID does not exist"
Oct 30 22:36:50.728572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.728537    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:36:50.778006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.777972    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-9a5623-openshift-default-5455ff6b5c-g2zf4"]
Oct 30 22:36:50.810463 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:36:50.810377    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:36:50.943940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.943909    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" path="/var/lib/kubelet/pods/7ab871ca-51ca-4f5f-8069-82c09968709e/volumes"
Oct 30 22:36:50.944429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:50.944414    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" path="/var/lib/kubelet/pods/dc329956-0457-467a-a679-7ee12ae8fcd8/volumes"
Oct 30 22:36:59.942548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:36:59.942515    4882 scope.go:122] "RemoveContainer" containerID="5c100462724b076ac023de09a4f3c46ee5b9bbadfa1907f7f3af58b90c84b278"
Oct 30 22:37:20.812266 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:37:20.812141    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:37:32.105854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:37:32.105810    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:37:50.808354 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:37:50.808317    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:37:57.489676 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:37:57.489640    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:37:57.509938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:37:57.509909    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:38:15.091815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:38:15.091728    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:38:20.809225 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:38:20.809183    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:38:50.808831 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:38:50.808789    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:39:15.183041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:15.183000    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:39:20.808981 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:39:20.808940    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:39:40.001287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:40.001203    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:39:40.001843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:40.001409    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" containerID="cri-o://9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8" gracePeriod=30
Oct 30 22:39:42.540902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:42.540862    4882 patch_prober.go:29] interesting pod/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:39:42.541324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:42.540943    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:39:45.265563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.265498    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:39:45.360704 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360658    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.360930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360735    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.360930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360764    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.360930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360787    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.360930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360821    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.360930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360894    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360939    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.360956    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361012    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361091    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\") pod \"496e5151-612d-4e8f-bc2d-e6bd068743d7\" (UID: \"496e5151-612d-4e8f-bc2d-e6bd068743d7\") "
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361136    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.361240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361165    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.361568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361256    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:39:45.361568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361404    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.361770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361746    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.361845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361773    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.361845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361790    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.361845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361806    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/496e5151-612d-4e8f-bc2d-e6bd068743d7-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.361845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.361820    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.363218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.363180    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:45.363218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.363210    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:39:45.363410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.363383    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:39:45.363476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.363460    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f" pod "496e5151-612d-4e8f-bc2d-e6bd068743d7" (UID: "496e5151-612d-4e8f-bc2d-e6bd068743d7"). InnerVolumeSpecName "kube-api-access-wf26f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:39:45.463006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.462920    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.463006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.462952    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.463006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.462965    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf26f\" (UniqueName: \"kubernetes.io/projected/496e5151-612d-4e8f-bc2d-e6bd068743d7-kube-api-access-wf26f\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.463006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.462979    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/496e5151-612d-4e8f-bc2d-e6bd068743d7-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:39:45.642991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.642957    4882 generic.go:356] "Generic (PLEG): container finished" podID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerID="9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8" exitCode=0
Oct 30 22:39:45.643511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.643014    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"
Oct 30 22:39:45.643511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.643023    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" event={"ID":"496e5151-612d-4e8f-bc2d-e6bd068743d7","Type":"ContainerDied","Data":"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"}
Oct 30 22:39:45.643511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.643091    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt" event={"ID":"496e5151-612d-4e8f-bc2d-e6bd068743d7","Type":"ContainerDied","Data":"0eea4cf6dac9d81f53f0ecfb52e1a83b369a99311125c0b8721563e56967c498"}
Oct 30 22:39:45.643511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.643125    4882 scope.go:122] "RemoveContainer" containerID="9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"
Oct 30 22:39:45.654059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.654033    4882 scope.go:122] "RemoveContainer" containerID="9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"
Oct 30 22:39:45.654388 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:39:45.654365    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8\": container with ID starting with 9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8 not found: ID does not exist" containerID="9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"
Oct 30 22:39:45.654458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.654398    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8"} err="failed to get container status \"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8\": rpc error: code = NotFound desc = could not find container \"9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8\": container with ID starting with 9c6410ef0bccfbaaa041d6eebf538a384052796e2a57298a581e073c98747cd8 not found: ID does not exist"
Oct 30 22:39:45.811841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.811803    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:39:45.814728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:45.814703    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-4d14e0-openshift-default-649bbfdc7f-k5brt"]
Oct 30 22:39:46.943419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:39:46.943381    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" path="/var/lib/kubelet/pods/496e5151-612d-4e8f-bc2d-e6bd068743d7/volumes"
Oct 30 22:39:50.808962 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:39:50.808909    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:40:04.874841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.874798    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"]
Oct 30 22:40:04.875392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875377    4882 cpu_manager.go:414] "containerMap: removing container" podUID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" containerName="payload-pre-processing"
Oct 30 22:40:04.875442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875393    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" containerName="payload-pre-processing"
Oct 30 22:40:04.875442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875414    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1e43950e-c523-4b59-8fab-45371d2970c6" containerName="main"
Oct 30 22:40:04.875442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875420    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e43950e-c523-4b59-8fab-45371d2970c6" containerName="main"
Oct 30 22:40:04.875442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875433    4882 cpu_manager.go:414] "containerMap: removing container" podUID="794d79d8-92c3-4358-a276-79f251a21db7" containerName="revoke-keys"
Oct 30 22:40:04.875442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875439    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794d79d8-92c3-4358-a276-79f251a21db7" containerName="revoke-keys"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875455    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875460    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875478    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875484    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875491    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875495    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875510    4882 cpu_manager.go:414] "containerMap: removing container" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875514    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875522    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875527    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875534    4882 cpu_manager.go:414] "containerMap: removing container" podUID="00ddcc84-9394-4cb9-a963-2fa624021fd2" containerName="payload-processing"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875539    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00ddcc84-9394-4cb9-a963-2fa624021fd2" containerName="payload-processing"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875642    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="496e5151-612d-4e8f-bc2d-e6bd068743d7" containerName="istio-proxy"
Oct 30 22:40:04.875649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875652    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ab871ca-51ca-4f5f-8069-82c09968709e" containerName="istio-proxy"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875661    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="00ddcc84-9394-4cb9-a963-2fa624021fd2" containerName="payload-processing"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875668    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc1fe04a-04cd-4435-8ba8-2692b5dba62c" containerName="payload-pre-processing"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875675    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e43950e-c523-4b59-8fab-45371d2970c6" containerName="main"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875681    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="794d79d8-92c3-4358-a276-79f251a21db7" containerName="revoke-keys"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875688    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f95518bf-37c2-446d-a964-aaa56b988b0b" containerName="istio-proxy"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875695    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1aa1e08-8443-4342-bd1c-1dd8bc60dcdd" containerName="istio-proxy"
Oct 30 22:40:04.876094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.875700    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc329956-0457-467a-a679-7ee12ae8fcd8" containerName="istio-proxy"
Oct 30 22:40:04.879094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.879077    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:04.933023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.932988    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"]
Oct 30 22:40:04.980291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.980259    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:04.980471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.980364    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:40:04.995391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.995355    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqw8l\" (UniqueName: \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:04.995575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.995547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:04.995661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:04.995613    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.096763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.096722    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.096959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.096793    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.096959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.096868    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqw8l\" (UniqueName: \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.097649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.097621    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.099425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.099396    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.159122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.159044    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqw8l\" (UniqueName: \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\") pod \"maas-api-revoke-sub-c9553a0b-78kh9\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.193655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.193611    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:05.567239 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:05.567196    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4601e620_e922_444f_90b6_4c435ec9a1ca.slice/crio-64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571 WatchSource:0}: Error finding container 64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571: Status 404 returned error can't find the container with id 64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571
Oct 30 22:40:05.569384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.569344    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"]
Oct 30 22:40:05.778400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:05.778364    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9" event={"ID":"4601e620-e922-444f-90b6-4c435ec9a1ca","Type":"ContainerStarted","Data":"64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571"}
Oct 30 22:40:06.786625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:06.786525    4882 generic.go:356] "Generic (PLEG): container finished" podID="4601e620-e922-444f-90b6-4c435ec9a1ca" containerID="636b40eaaded917aafd3aafa18dcb7a7fe5f356dd7f314be64e7ae84a0ae8f0a" exitCode=0
Oct 30 22:40:06.787009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:06.786617    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9" event={"ID":"4601e620-e922-444f-90b6-4c435ec9a1ca","Type":"ContainerDied","Data":"636b40eaaded917aafd3aafa18dcb7a7fe5f356dd7f314be64e7ae84a0ae8f0a"}
Oct 30 22:40:07.954242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:07.954213    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:08.031705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.031670    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\" (UniqueName: \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\") pod \"4601e620-e922-444f-90b6-4c435ec9a1ca\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") "
Oct 30 22:40:08.031887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.031849    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\") pod \"4601e620-e922-444f-90b6-4c435ec9a1ca\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") "
Oct 30 22:40:08.031887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.031876    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\") pod \"4601e620-e922-444f-90b6-4c435ec9a1ca\" (UID: \"4601e620-e922-444f-90b6-4c435ec9a1ca\") "
Oct 30 22:40:08.032224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.032197    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca" pod "4601e620-e922-444f-90b6-4c435ec9a1ca" (UID: "4601e620-e922-444f-90b6-4c435ec9a1ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:08.033860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.033841    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l" pod "4601e620-e922-444f-90b6-4c435ec9a1ca" (UID: "4601e620-e922-444f-90b6-4c435ec9a1ca"). InnerVolumeSpecName "kube-api-access-gqw8l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:08.033988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.033967    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca" pod "4601e620-e922-444f-90b6-4c435ec9a1ca" (UID: "4601e620-e922-444f-90b6-4c435ec9a1ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:08.132770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.132723    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:08.132770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.132768    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4601e620-e922-444f-90b6-4c435ec9a1ca-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:08.132770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.132778    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqw8l\" (UniqueName: \"kubernetes.io/projected/4601e620-e922-444f-90b6-4c435ec9a1ca-kube-api-access-gqw8l\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:08.799121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.799089    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"
Oct 30 22:40:08.799121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.799107    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9" event={"ID":"4601e620-e922-444f-90b6-4c435ec9a1ca","Type":"ContainerDied","Data":"64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571"}
Oct 30 22:40:08.799392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:08.799170    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64854faa30be5ac01f04cb80b6861cdf5a1f7642cc99f6f62a6772516f84b571"
Oct 30 22:40:15.594001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.589802    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"]
Oct 30 22:40:15.594001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.591219    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4601e620-e922-444f-90b6-4c435ec9a1ca" containerName="revoke-keys"
Oct 30 22:40:15.594001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.591247    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4601e620-e922-444f-90b6-4c435ec9a1ca" containerName="revoke-keys"
Oct 30 22:40:15.594001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.591520    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4601e620-e922-444f-90b6-4c435ec9a1ca" containerName="revoke-keys"
Oct 30 22:40:15.596902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.596870    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.620340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.620310    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.620525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.620349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpxmh\" (UniqueName: \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.620633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.620609    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.647390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.647347    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"]
Oct 30 22:40:15.692953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.692920    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:15.693109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.692966    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:40:15.721718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.721687    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.721920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.721727    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpxmh\" (UniqueName: \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.721920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.721840    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.722454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.722428    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.724115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.724091    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.784028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.783997    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpxmh\" (UniqueName: \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\") pod \"maas-api-revoke-sub-b25a8634-jjxms\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:15.914637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:15.914523    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:16.289307 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:16.289279    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79229eb7_ee79_4b8a_8add_178dc1648b57.slice/crio-fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b WatchSource:0}: Error finding container fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b: Status 404 returned error can't find the container with id fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b
Oct 30 22:40:16.293210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:16.293182    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"]
Oct 30 22:40:16.848254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:16.848223    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms" event={"ID":"79229eb7-ee79-4b8a-8add-178dc1648b57","Type":"ContainerStarted","Data":"6fcf4c9254c06515a8efc3ea90156ae4e7963c213511f98061e4da9728c1783f"}
Oct 30 22:40:16.848779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:16.848277    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms" event={"ID":"79229eb7-ee79-4b8a-8add-178dc1648b57","Type":"ContainerStarted","Data":"fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b"}
Oct 30 22:40:16.971155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:16.971088    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms" podStartSLOduration=1.97096633 podStartE2EDuration="1.97096633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:16.966630777 +0000 UTC m=+3740.139955206" watchObservedRunningTime="2026-10-30 22:40:16.97096633 +0000 UTC m=+3740.144290758"
Oct 30 22:40:17.855743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:17.855709    4882 generic.go:356] "Generic (PLEG): container finished" podID="79229eb7-ee79-4b8a-8add-178dc1648b57" containerID="6fcf4c9254c06515a8efc3ea90156ae4e7963c213511f98061e4da9728c1783f" exitCode=0
Oct 30 22:40:17.856181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:17.855799    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms" event={"ID":"79229eb7-ee79-4b8a-8add-178dc1648b57","Type":"ContainerDied","Data":"6fcf4c9254c06515a8efc3ea90156ae4e7963c213511f98061e4da9728c1783f"}
Oct 30 22:40:19.028901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.028873    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:19.167987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.167884    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\") pod \"79229eb7-ee79-4b8a-8add-178dc1648b57\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") "
Oct 30 22:40:19.168171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.167998    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\" (UniqueName: \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\") pod \"79229eb7-ee79-4b8a-8add-178dc1648b57\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") "
Oct 30 22:40:19.168171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.168069    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\") pod \"79229eb7-ee79-4b8a-8add-178dc1648b57\" (UID: \"79229eb7-ee79-4b8a-8add-178dc1648b57\") "
Oct 30 22:40:19.168526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.168496    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca" pod "79229eb7-ee79-4b8a-8add-178dc1648b57" (UID: "79229eb7-ee79-4b8a-8add-178dc1648b57"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:19.168845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.168817    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.170348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.170324    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh" pod "79229eb7-ee79-4b8a-8add-178dc1648b57" (UID: "79229eb7-ee79-4b8a-8add-178dc1648b57"). InnerVolumeSpecName "kube-api-access-rpxmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:19.170520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.170502    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca" pod "79229eb7-ee79-4b8a-8add-178dc1648b57" (UID: "79229eb7-ee79-4b8a-8add-178dc1648b57"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:19.270266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.270232    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79229eb7-ee79-4b8a-8add-178dc1648b57-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.270266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.270263    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpxmh\" (UniqueName: \"kubernetes.io/projected/79229eb7-ee79-4b8a-8add-178dc1648b57-kube-api-access-rpxmh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:19.872062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.872033    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"
Oct 30 22:40:19.872253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.872069    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms" event={"ID":"79229eb7-ee79-4b8a-8add-178dc1648b57","Type":"ContainerDied","Data":"fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b"}
Oct 30 22:40:19.872253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:19.872145    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe03dd6d80074f088ad6d29c502b5d305e4c0a5d2b74afa900d993076b5e076b"
Oct 30 22:40:20.808502 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:20.808465    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:40:29.395225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:29.395181    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:40:30.856079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.856043    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:40:30.964275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.964237    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7658-9bc8f7"]
Oct 30 22:40:30.964818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.964800    4882 cpu_manager.go:414] "containerMap: removing container" podUID="79229eb7-ee79-4b8a-8add-178dc1648b57" containerName="revoke-keys"
Oct 30 22:40:30.964818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.964816    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79229eb7-ee79-4b8a-8add-178dc1648b57" containerName="revoke-keys"
Oct 30 22:40:30.965025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.964941    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="79229eb7-ee79-4b8a-8add-178dc1648b57" containerName="revoke-keys"
Oct 30 22:40:30.969615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:30.969562    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:31.019978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.019935    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-9bc8f7"]
Oct 30 22:40:31.023365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.023329    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jppg\""
Oct 30 22:40:31.112642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.112524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz4pd\" (UniqueName: \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\") pod \"test-curl-7658-9bc8f7\" (UID: \"e7af919a-2044-4233-aebe-52654dc7dc7b\") " pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:31.213553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.213520    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz4pd\" (UniqueName: \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\") pod \"test-curl-7658-9bc8f7\" (UID: \"e7af919a-2044-4233-aebe-52654dc7dc7b\") " pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:31.275998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.275960    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz4pd\" (UniqueName: \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\") pod \"test-curl-7658-9bc8f7\" (UID: \"e7af919a-2044-4233-aebe-52654dc7dc7b\") " pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:31.285699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.285665    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:31.660701 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:31.660663    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode7af919a_2044_4233_aebe_52654dc7dc7b.slice/crio-9d11731058a30c924a0a9e207e95bb07097908a88fe19d14973b3417018a6477 WatchSource:0}: Error finding container 9d11731058a30c924a0a9e207e95bb07097908a88fe19d14973b3417018a6477: Status 404 returned error can't find the container with id 9d11731058a30c924a0a9e207e95bb07097908a88fe19d14973b3417018a6477
Oct 30 22:40:31.663068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.663045    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-9bc8f7"]
Oct 30 22:40:31.950739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.950650    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-9bc8f7" event={"ID":"e7af919a-2044-4233-aebe-52654dc7dc7b","Type":"ContainerStarted","Data":"ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1"}
Oct 30 22:40:31.950739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:31.950694    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-9bc8f7" event={"ID":"e7af919a-2044-4233-aebe-52654dc7dc7b","Type":"ContainerStarted","Data":"9d11731058a30c924a0a9e207e95bb07097908a88fe19d14973b3417018a6477"}
Oct 30 22:40:33.534852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.534786    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7658-9bc8f7" podStartSLOduration=3.534770011 podStartE2EDuration="3.534770011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:32.067811712 +0000 UTC m=+3755.241136139" watchObservedRunningTime="2026-10-30 22:40:33.534770011 +0000 UTC m=+3756.708094438"
Oct 30 22:40:33.536471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.536441    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7658-9bc8f7"]
Oct 30 22:40:33.544611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.544556    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7658-9bc8f7"]
Oct 30 22:40:33.595401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.595360    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:40:33.702313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.702278    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7658-9d8cc3"]
Oct 30 22:40:33.702879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.702863    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" containerName="test-curl-7658-9bc8f7"
Oct 30 22:40:33.702935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.702881    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" containerName="test-curl-7658-9bc8f7"
Oct 30 22:40:33.703035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.703025    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" containerName="test-curl-7658-9bc8f7"
Oct 30 22:40:33.707753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.707723    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:33.760851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.760819    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-9d8cc3"]
Oct 30 22:40:33.845060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.845018    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hws4w\" (UniqueName: \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\") pod \"test-curl-7658-9d8cc3\" (UID: \"4b12f443-027d-43cf-bd30-710479d667c9\") " pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:33.946248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.946212    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hws4w\" (UniqueName: \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\") pod \"test-curl-7658-9d8cc3\" (UID: \"4b12f443-027d-43cf-bd30-710479d667c9\") " pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:33.963325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:33.963285    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7658-9bc8f7" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" containerName="test-curl-7658-9bc8f7" containerID="cri-o://ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1" gracePeriod=2
Oct 30 22:40:34.008581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:34.008543    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hws4w\" (UniqueName: \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\") pod \"test-curl-7658-9d8cc3\" (UID: \"4b12f443-027d-43cf-bd30-710479d667c9\") " pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:34.015327 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:34.015289    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9bc8f7\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:34.026749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:34.026716    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:34.419294 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:34.419264    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4b12f443_027d_43cf_bd30_710479d667c9.slice/crio-4641c92bc51de84d5dc8930801c111d7a0230455256ec1e878b238d360ef4588 WatchSource:0}: Error finding container 4641c92bc51de84d5dc8930801c111d7a0230455256ec1e878b238d360ef4588: Status 404 returned error can't find the container with id 4641c92bc51de84d5dc8930801c111d7a0230455256ec1e878b238d360ef4588
Oct 30 22:40:34.421628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:34.421601    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-9d8cc3"]
Oct 30 22:40:34.970629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:34.970564    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-9d8cc3" event={"ID":"4b12f443-027d-43cf-bd30-710479d667c9","Type":"ContainerStarted","Data":"0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070"}
Oct 30 22:40:34.971032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:34.970639    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-9d8cc3" event={"ID":"4b12f443-027d-43cf-bd30-710479d667c9","Type":"ContainerStarted","Data":"4641c92bc51de84d5dc8930801c111d7a0230455256ec1e878b238d360ef4588"}
Oct 30 22:40:35.085977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:35.085913    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7658-9d8cc3" podStartSLOduration=2.08589607 podStartE2EDuration="2.08589607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:35.085393567 +0000 UTC m=+3758.258717994" watchObservedRunningTime="2026-10-30 22:40:35.08589607 +0000 UTC m=+3758.259220497"
Oct 30 22:40:36.212367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.212342    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:36.275744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.275662    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\" (UniqueName: \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\") pod \"e7af919a-2044-4233-aebe-52654dc7dc7b\" (UID: \"e7af919a-2044-4233-aebe-52654dc7dc7b\") "
Oct 30 22:40:36.277837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.277816    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd" pod "e7af919a-2044-4233-aebe-52654dc7dc7b" (UID: "e7af919a-2044-4233-aebe-52654dc7dc7b"). InnerVolumeSpecName "kube-api-access-cz4pd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:36.376994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.376954    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cz4pd\" (UniqueName: \"kubernetes.io/projected/e7af919a-2044-4233-aebe-52654dc7dc7b-kube-api-access-cz4pd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:36.683344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.683304    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7658-9d8cc3"]
Oct 30 22:40:36.686928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.686895    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7658-9d8cc3"]
Oct 30 22:40:36.844681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.844648    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7658-8e7216"]
Oct 30 22:40:36.845471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.845450    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4b12f443-027d-43cf-bd30-710479d667c9" containerName="test-curl-7658-9d8cc3"
Oct 30 22:40:36.845524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.845473    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b12f443-027d-43cf-bd30-710479d667c9" containerName="test-curl-7658-9d8cc3"
Oct 30 22:40:36.845637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.845625    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b12f443-027d-43cf-bd30-710479d667c9" containerName="test-curl-7658-9d8cc3"
Oct 30 22:40:36.849082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.849065    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:36.883238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.883201    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp2rc\" (UniqueName: \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\") pod \"test-curl-7658-8e7216\" (UID: \"47f992fd-076d-4bce-87a7-b6155d360f45\") " pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:36.901793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.901760    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-8e7216"]
Oct 30 22:40:36.944192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.944116    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" path="/var/lib/kubelet/pods/e7af919a-2044-4233-aebe-52654dc7dc7b/volumes"
Oct 30 22:40:36.984499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.984468    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp2rc\" (UniqueName: \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\") pod \"test-curl-7658-8e7216\" (UID: \"47f992fd-076d-4bce-87a7-b6155d360f45\") " pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:36.985822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.985787    4882 generic.go:356] "Generic (PLEG): container finished" podID="e7af919a-2044-4233-aebe-52654dc7dc7b" containerID="ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1" exitCode=137
Oct 30 22:40:36.985984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.985829    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:36.985984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.985918    4882 scope.go:122] "RemoveContainer" containerID="ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1"
Oct 30 22:40:36.986123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.986039    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7658-9d8cc3" podUID="4b12f443-027d-43cf-bd30-710479d667c9" containerName="test-curl-7658-9d8cc3" containerID="cri-o://0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070" gracePeriod=2
Oct 30 22:40:36.995849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.995830    4882 scope.go:122] "RemoveContainer" containerID="ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1"
Oct 30 22:40:36.996148 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:36.996128    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1\": container with ID starting with ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1 not found: ID does not exist" containerID="ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1"
Oct 30 22:40:36.996197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:36.996158    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1"} err="failed to get container status \"ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1\": rpc error: code = NotFound desc = could not find container \"ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1\": container with ID starting with ec19bc25d276c3a6c4dda0a52b8fc33cdbb3dbc002b3a26dacd84c9ae2f4aaf1 not found: ID does not exist"
Oct 30 22:40:37.046390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:37.046355    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp2rc\" (UniqueName: \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\") pod \"test-curl-7658-8e7216\" (UID: \"47f992fd-076d-4bce-87a7-b6155d360f45\") " pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:37.072301 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:37.072262    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:37.124213 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:37.124174    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9bc8f7\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:37.161369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:37.161341    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:37.535135 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:37.535094    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47f992fd_076d_4bce_87a7_b6155d360f45.slice/crio-8da87b06f72575f514d1ed5ab3960ca8dff2124a04506b13596fbc2e5759b33d WatchSource:0}: Error finding container 8da87b06f72575f514d1ed5ab3960ca8dff2124a04506b13596fbc2e5759b33d: Status 404 returned error can't find the container with id 8da87b06f72575f514d1ed5ab3960ca8dff2124a04506b13596fbc2e5759b33d
Oct 30 22:40:37.537634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:37.537607    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-8e7216"]
Oct 30 22:40:37.993498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:37.993451    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-8e7216" event={"ID":"47f992fd-076d-4bce-87a7-b6155d360f45","Type":"ContainerStarted","Data":"7e036ec14b68394300d28ff6c701c260dfc611348b15a7a94292167ebed35234"}
Oct 30 22:40:37.993727 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:37.993506    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-8e7216" event={"ID":"47f992fd-076d-4bce-87a7-b6155d360f45","Type":"ContainerStarted","Data":"8da87b06f72575f514d1ed5ab3960ca8dff2124a04506b13596fbc2e5759b33d"}
Oct 30 22:40:38.045468 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:38.045419    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:38.097207 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:38.097151    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9bc8f7\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="e7af919a-2044-4233-aebe-52654dc7dc7b" pod="opendatahub/test-curl-7658-9bc8f7"
Oct 30 22:40:38.213341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:38.213280    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7658-8e7216" podStartSLOduration=2.213259644 podStartE2EDuration="2.213259644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:38.211384579 +0000 UTC m=+3761.384709007" watchObservedRunningTime="2026-10-30 22:40:38.213259644 +0000 UTC m=+3761.386584071"
Oct 30 22:40:39.001458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.001426    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"]
Oct 30 22:40:39.009756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.009728    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.063135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.063100    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"]
Oct 30 22:40:39.106301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.106260    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:39.106513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.106398    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:40:39.111889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.111867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gd4hw\" (UniqueName: \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.111970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.111939    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.112045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.112029    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.213254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.213221    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gd4hw\" (UniqueName: \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.213418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.213274    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.213418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.213331    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.214081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.214055    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.215859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.215834    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.243363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.243292    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:39.275641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.275571    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gd4hw\" (UniqueName: \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\") pod \"maas-api-revoke-sub-3a4b4289-rtvpl\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.295276 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:39.295239    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:39.314472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.314438    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\" (UniqueName: \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\") pod \"4b12f443-027d-43cf-bd30-710479d667c9\" (UID: \"4b12f443-027d-43cf-bd30-710479d667c9\") "
Oct 30 22:40:39.316668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.316640    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w" pod "4b12f443-027d-43cf-bd30-710479d667c9" (UID: "4b12f443-027d-43cf-bd30-710479d667c9"). InnerVolumeSpecName "kube-api-access-hws4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:39.364413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.364374    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:39.415992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.415946    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hws4w\" (UniqueName: \"kubernetes.io/projected/4b12f443-027d-43cf-bd30-710479d667c9-kube-api-access-hws4w\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:39.640859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.640820    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7658-8e7216"]
Oct 30 22:40:39.645452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.645416    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7658-8e7216"]
Oct 30 22:40:39.743679 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:39.743649    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93f04dbb_9074_4ae8_8da1_41d7be82b0c3.slice/crio-c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc WatchSource:0}: Error finding container c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc: Status 404 returned error can't find the container with id c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc
Oct 30 22:40:39.746232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.746208    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"]
Oct 30 22:40:39.822151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.822108    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7658-11fc4d"]
Oct 30 22:40:39.822726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.822711    4882 cpu_manager.go:414] "containerMap: removing container" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" containerName="test-curl-7658-8e7216"
Oct 30 22:40:39.822791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.822729    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" containerName="test-curl-7658-8e7216"
Oct 30 22:40:39.822851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.822842    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" containerName="test-curl-7658-8e7216"
Oct 30 22:40:39.827379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.827358    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:39.878137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.878101    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-11fc4d"]
Oct 30 22:40:39.879196 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:39.879162    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:39.922171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:39.922093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbsfb\" (UniqueName: \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\") pod \"test-curl-7658-11fc4d\" (UID: \"b41e0e80-c6b4-4f07-a650-2d0b97692154\") " pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:40.009921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.009887    4882 generic.go:356] "Generic (PLEG): container finished" podID="4b12f443-027d-43cf-bd30-710479d667c9" containerID="0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070" exitCode=137
Oct 30 22:40:40.010389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.009933    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:40.010389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.010000    4882 scope.go:122] "RemoveContainer" containerID="0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070"
Oct 30 22:40:40.011796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.011746    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl" event={"ID":"93f04dbb-9074-4ae8-8da1-41d7be82b0c3","Type":"ContainerStarted","Data":"c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc"}
Oct 30 22:40:40.011968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.011946    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7658-8e7216" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" containerName="test-curl-7658-8e7216" containerID="cri-o://7e036ec14b68394300d28ff6c701c260dfc611348b15a7a94292167ebed35234" gracePeriod=2
Oct 30 22:40:40.021056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.021028    4882 scope.go:122] "RemoveContainer" containerID="0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070"
Oct 30 22:40:40.021368 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:40.021349    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070\": container with ID starting with 0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070 not found: ID does not exist" containerID="0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070"
Oct 30 22:40:40.021427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.021378    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070"} err="failed to get container status \"0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070\": rpc error: code = NotFound desc = could not find container \"0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070\": container with ID starting with 0f47cf7fff95f3b6092c35560d81e3ebec96c6e4fbce0632c28e0fc09971f070 not found: ID does not exist"
Oct 30 22:40:40.023062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.023041    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbsfb\" (UniqueName: \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\") pod \"test-curl-7658-11fc4d\" (UID: \"b41e0e80-c6b4-4f07-a650-2d0b97692154\") " pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:40.061889 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:40.061852    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:40.084855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.084820    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbsfb\" (UniqueName: \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\") pod \"test-curl-7658-11fc4d\" (UID: \"b41e0e80-c6b4-4f07-a650-2d0b97692154\") " pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:40.113539 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:40.113495    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-8e7216\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:40.139492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.139456    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:40.165441 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:40.165408    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-9d8cc3\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="4b12f443-027d-43cf-bd30-710479d667c9" pod="opendatahub/test-curl-7658-9d8cc3"
Oct 30 22:40:40.511467 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:40.511433    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb41e0e80_c6b4_4f07_a650_2d0b97692154.slice/crio-f735b23a0745752a9acae1675c46ad3b3b6ab37f37196024a088830b483cadcc WatchSource:0}: Error finding container f735b23a0745752a9acae1675c46ad3b3b6ab37f37196024a088830b483cadcc: Status 404 returned error can't find the container with id f735b23a0745752a9acae1675c46ad3b3b6ab37f37196024a088830b483cadcc
Oct 30 22:40:40.524834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.522056    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7658-11fc4d"]
Oct 30 22:40:40.943420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:40.943386    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b12f443-027d-43cf-bd30-710479d667c9" path="/var/lib/kubelet/pods/4b12f443-027d-43cf-bd30-710479d667c9/volumes"
Oct 30 22:40:41.021718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:41.021680    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-11fc4d" event={"ID":"b41e0e80-c6b4-4f07-a650-2d0b97692154","Type":"ContainerStarted","Data":"29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c"}
Oct 30 22:40:41.022210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:41.021735    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7658-11fc4d" event={"ID":"b41e0e80-c6b4-4f07-a650-2d0b97692154","Type":"ContainerStarted","Data":"f735b23a0745752a9acae1675c46ad3b3b6ab37f37196024a088830b483cadcc"}
Oct 30 22:40:41.023711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:41.023679    4882 generic.go:356] "Generic (PLEG): container finished" podID="93f04dbb-9074-4ae8-8da1-41d7be82b0c3" containerID="4951b04eafed3791d01ae8527f00d4625d56402bbeb1190ae59eca668f832f31" exitCode=0
Oct 30 22:40:41.023850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:41.023765    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl" event={"ID":"93f04dbb-9074-4ae8-8da1-41d7be82b0c3","Type":"ContainerDied","Data":"4951b04eafed3791d01ae8527f00d4625d56402bbeb1190ae59eca668f832f31"}
Oct 30 22:40:41.140597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:41.140536    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7658-11fc4d" podStartSLOduration=2.140519859 podStartE2EDuration="2.140519859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:41.138543681 +0000 UTC m=+3764.311868109" watchObservedRunningTime="2026-10-30 22:40:41.140519859 +0000 UTC m=+3764.313844286"
Oct 30 22:40:42.034748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.034691    4882 generic.go:356] "Generic (PLEG): container finished" podID="47f992fd-076d-4bce-87a7-b6155d360f45" containerID="7e036ec14b68394300d28ff6c701c260dfc611348b15a7a94292167ebed35234" exitCode=137
Oct 30 22:40:42.197324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.197290    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:42.231903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.231877    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:42.248150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.248112    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\") pod \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") "
Oct 30 22:40:42.248317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.248169    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\") pod \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") "
Oct 30 22:40:42.248317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.248233    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\" (UniqueName: \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\") pod \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\" (UID: \"93f04dbb-9074-4ae8-8da1-41d7be82b0c3\") "
Oct 30 22:40:42.248565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.248524    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca" pod "93f04dbb-9074-4ae8-8da1-41d7be82b0c3" (UID: "93f04dbb-9074-4ae8-8da1-41d7be82b0c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:42.250397 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.250370    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca" pod "93f04dbb-9074-4ae8-8da1-41d7be82b0c3" (UID: "93f04dbb-9074-4ae8-8da1-41d7be82b0c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:42.250499 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.250437    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw" pod "93f04dbb-9074-4ae8-8da1-41d7be82b0c3" (UID: "93f04dbb-9074-4ae8-8da1-41d7be82b0c3"). InnerVolumeSpecName "kube-api-access-gd4hw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:42.349876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.349837    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\" (UniqueName: \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\") pod \"47f992fd-076d-4bce-87a7-b6155d360f45\" (UID: \"47f992fd-076d-4bce-87a7-b6155d360f45\") "
Oct 30 22:40:42.350439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.350416    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:42.350563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.350440    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:42.350563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.350457    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gd4hw\" (UniqueName: \"kubernetes.io/projected/93f04dbb-9074-4ae8-8da1-41d7be82b0c3-kube-api-access-gd4hw\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:42.352001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.351975    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc" pod "47f992fd-076d-4bce-87a7-b6155d360f45" (UID: "47f992fd-076d-4bce-87a7-b6155d360f45"). InnerVolumeSpecName "kube-api-access-lp2rc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:42.451659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.451577    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp2rc\" (UniqueName: \"kubernetes.io/projected/47f992fd-076d-4bce-87a7-b6155d360f45-kube-api-access-lp2rc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:42.944675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:42.944640    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47f992fd-076d-4bce-87a7-b6155d360f45" path="/var/lib/kubelet/pods/47f992fd-076d-4bce-87a7-b6155d360f45/volumes"
Oct 30 22:40:43.042597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:43.042552    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"
Oct 30 22:40:43.043034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:43.042559    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl" event={"ID":"93f04dbb-9074-4ae8-8da1-41d7be82b0c3","Type":"ContainerDied","Data":"c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc"}
Oct 30 22:40:43.043034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:43.042657    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6fe40fdcd81fc464ce2226a625d843423fa46521a8360f011ef04f9e90b06bc"
Oct 30 22:40:43.044125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:43.044093    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-8e7216"
Oct 30 22:40:43.044250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:43.044148    4882 scope.go:122] "RemoveContainer" containerID="7e036ec14b68394300d28ff6c701c260dfc611348b15a7a94292167ebed35234"
Oct 30 22:40:44.424490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.424451    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" podUID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" containerName="main" containerID="cri-o://8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3" gracePeriod=45
Oct 30 22:40:44.696965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.696938    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:40:44.774871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.774820    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.774871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.774878    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.775111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775008    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\" (UniqueName: \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.775111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775043    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.775238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775131    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\" (UniqueName: \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.775238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775187    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\") pod \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\" (UID: \"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da\") "
Oct 30 22:40:44.775238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775166    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:44.775238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:44.775472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775370    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:44.775680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775656    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:44.775680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775679    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:44.775867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.775694    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:44.777396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.777368    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:44.777481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.777403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:44.777481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.777443    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4" pod "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" (UID: "61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da"). InnerVolumeSpecName "kube-api-access-7fmt4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:44.876508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.876472    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:44.876508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.876508    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:44.876508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:44.876522    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fmt4\" (UniqueName: \"kubernetes.io/projected/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da-kube-api-access-7fmt4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:45.066336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.066298    4882 generic.go:356] "Generic (PLEG): container finished" podID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" containerID="8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3" exitCode=0
Oct 30 22:40:45.066541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.066354    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"
Oct 30 22:40:45.066541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.066386    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" event={"ID":"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da","Type":"ContainerDied","Data":"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"}
Oct 30 22:40:45.066541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.066432    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4" event={"ID":"61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da","Type":"ContainerDied","Data":"9362bd13bc1385909d7f88072d4f721548677610a8f726291a1057b3913b7e1e"}
Oct 30 22:40:45.066541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.066456    4882 scope.go:122] "RemoveContainer" containerID="8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"
Oct 30 22:40:45.076005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.075980    4882 scope.go:122] "RemoveContainer" containerID="8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"
Oct 30 22:40:45.076326 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:45.076304    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3\": container with ID starting with 8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3 not found: ID does not exist" containerID="8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"
Oct 30 22:40:45.076380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.076336    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3"} err="failed to get container status \"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3\": rpc error: code = NotFound desc = could not find container \"8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3\": container with ID starting with 8cd6fb4cd8071b6f997b1f483900997f4cf0c8ef5e8ba98092c5e740498f5fc3 not found: ID does not exist"
Oct 30 22:40:45.234498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.234391    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:40:45.240183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.240157    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w2-5e884c/auth-test-model-5e884c-kserve-f8587685-8rkh4"]
Oct 30 22:40:45.877350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:45.877293    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerName="main" containerID="cri-o://b73a99a7ba3363e4f3597a27703e7be768f3875de225058be5c791d5d8d2bb9f" gracePeriod=45
Oct 30 22:40:46.044163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.044120    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerName="main" probeResult="failure" output="Get \"https://10.132.1.29:8000/ready\": dial tcp 10.132.1.29:8000: connect: connection refused"
Oct 30 22:40:46.076606 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.076543    4882 generic.go:356] "Generic (PLEG): container finished" podID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerID="b73a99a7ba3363e4f3597a27703e7be768f3875de225058be5c791d5d8d2bb9f" exitCode=0
Oct 30 22:40:46.076760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.076618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" event={"ID":"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113","Type":"ContainerDied","Data":"b73a99a7ba3363e4f3597a27703e7be768f3875de225058be5c791d5d8d2bb9f"}
Oct 30 22:40:46.149230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.149205    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:40:46.190835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.190786    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.190886    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.190932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\" (UniqueName: \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191083    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\" (UniqueName: \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191158    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:46.191198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191274    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\") pod \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\" (UID: \"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113\") "
Oct 30 22:40:46.191434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191393    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:46.191760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191730    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:46.191886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191817    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.191886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.191839    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.193177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.193154    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:40:46.193419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.193394    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "kube-api-access-xswdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:46.193568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.193542    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs" pod "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" (UID: "28a5c2a9-49b3-4122-8a4a-bae9c3b8c113"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:46.292643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.292603    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.292643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.292640    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.292643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.292650    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xswdv\" (UniqueName: \"kubernetes.io/projected/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-kube-api-access-xswdv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.292894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.292662    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:46.945851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:46.945813    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" path="/var/lib/kubelet/pods/61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da/volumes"
Oct 30 22:40:47.084273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:47.084241    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"
Oct 30 22:40:47.084486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:47.084241    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8" event={"ID":"28a5c2a9-49b3-4122-8a4a-bae9c3b8c113","Type":"ContainerDied","Data":"f94437605b504d8aae26ade18c270f55891571b78f786870f1f810952932eeda"}
Oct 30 22:40:47.084486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:47.084373    4882 scope.go:122] "RemoveContainer" containerID="b73a99a7ba3363e4f3597a27703e7be768f3875de225058be5c791d5d8d2bb9f"
Oct 30 22:40:47.252087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:47.252002    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:40:47.302183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:47.302145    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w2-5bb0b1/auth-test-model-5bb0b1-kserve-6557f76f77-7fnw8"]
Oct 30 22:40:48.943116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:48.943081    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" path="/var/lib/kubelet/pods/28a5c2a9-49b3-4122-8a4a-bae9c3b8c113/volumes"
Oct 30 22:40:49.745230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745188    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"]
Oct 30 22:40:49.745792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745773    4882 cpu_manager.go:414] "containerMap: removing container" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerName="main"
Oct 30 22:40:49.745792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745791    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerName="main"
Oct 30 22:40:49.745922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745813    4882 cpu_manager.go:414] "containerMap: removing container" podUID="93f04dbb-9074-4ae8-8da1-41d7be82b0c3" containerName="revoke-keys"
Oct 30 22:40:49.745922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745819    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93f04dbb-9074-4ae8-8da1-41d7be82b0c3" containerName="revoke-keys"
Oct 30 22:40:49.745922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745832    4882 cpu_manager.go:414] "containerMap: removing container" podUID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" containerName="main"
Oct 30 22:40:49.745922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745838    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" containerName="main"
Oct 30 22:40:49.746078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745958    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="93f04dbb-9074-4ae8-8da1-41d7be82b0c3" containerName="revoke-keys"
Oct 30 22:40:49.746078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745970    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d7d6fb-2bfd-406e-85d6-d5e2d2f0a3da" containerName="main"
Oct 30 22:40:49.746078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.745984    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="28a5c2a9-49b3-4122-8a4a-bae9c3b8c113" containerName="main"
Oct 30 22:40:49.750915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.750895    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.805955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.805909    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"]
Oct 30 22:40:49.832813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.832369    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrd2n\" (UniqueName: \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.832813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.832528    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.832813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.832646    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.852509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.852472    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:40:49.852748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.852731    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:40:49.933519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.933482    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrd2n\" (UniqueName: \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.933736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.933593    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.933736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.933648    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.934350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.934330    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.936431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.936401    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:49.996552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:49.996470    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrd2n\" (UniqueName: \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\") pod \"maas-api-revoke-sub-e8f1d2aa-4244g\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:50.064135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:50.064102    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:50.446463 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:40:50.446432    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod933df945_d492_4035_a607_ad6e72926d9a.slice/crio-6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66 WatchSource:0}: Error finding container 6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66: Status 404 returned error can't find the container with id 6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66
Oct 30 22:40:50.450298 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:50.450272    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"]
Oct 30 22:40:50.808698 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:50.808662    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:40:51.112690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:51.112652    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g" event={"ID":"933df945-d492-4035-a607-ad6e72926d9a","Type":"ContainerStarted","Data":"ed6b72ae849c7a5c1b431cbbb53ae01c5dd63aaa41e502c857fa8bacf8d4ffae"}
Oct 30 22:40:51.113151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:51.112707    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g" event={"ID":"933df945-d492-4035-a607-ad6e72926d9a","Type":"ContainerStarted","Data":"6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66"}
Oct 30 22:40:51.270901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:51.270840    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g" podStartSLOduration=2.270825112 podStartE2EDuration="2.270825112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:40:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:40:51.270390258 +0000 UTC m=+3774.443714687" watchObservedRunningTime="2026-10-30 22:40:51.270825112 +0000 UTC m=+3774.444149583"
Oct 30 22:40:52.120355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:52.120319    4882 generic.go:356] "Generic (PLEG): container finished" podID="933df945-d492-4035-a607-ad6e72926d9a" containerID="ed6b72ae849c7a5c1b431cbbb53ae01c5dd63aaa41e502c857fa8bacf8d4ffae" exitCode=0
Oct 30 22:40:52.120801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:52.120369    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g" event={"ID":"933df945-d492-4035-a607-ad6e72926d9a","Type":"ContainerDied","Data":"ed6b72ae849c7a5c1b431cbbb53ae01c5dd63aaa41e502c857fa8bacf8d4ffae"}
Oct 30 22:40:52.698447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:52.698413    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7658-11fc4d"]
Oct 30 22:40:52.698720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:52.698645    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7658-11fc4d" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" containerName="test-curl-7658-11fc4d" containerID="cri-o://29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c" gracePeriod=2
Oct 30 22:40:52.703545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:52.703514    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7658-11fc4d"]
Oct 30 22:40:52.750557 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:52.750511    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-11fc4d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:53.292808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.292783    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:53.372966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.372925    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\") pod \"933df945-d492-4035-a607-ad6e72926d9a\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") "
Oct 30 22:40:53.373155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.373014    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\" (UniqueName: \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\") pod \"933df945-d492-4035-a607-ad6e72926d9a\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") "
Oct 30 22:40:53.373155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.373154    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\") pod \"933df945-d492-4035-a607-ad6e72926d9a\" (UID: \"933df945-d492-4035-a607-ad6e72926d9a\") "
Oct 30 22:40:53.373523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.373486    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca" pod "933df945-d492-4035-a607-ad6e72926d9a" (UID: "933df945-d492-4035-a607-ad6e72926d9a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:40:53.375199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.375172    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n" pod "933df945-d492-4035-a607-ad6e72926d9a" (UID: "933df945-d492-4035-a607-ad6e72926d9a"). InnerVolumeSpecName "kube-api-access-rrd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:53.375199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.375179    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca" pod "933df945-d492-4035-a607-ad6e72926d9a" (UID: "933df945-d492-4035-a607-ad6e72926d9a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:40:53.474720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.474673    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/933df945-d492-4035-a607-ad6e72926d9a-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.474720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.474718    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/933df945-d492-4035-a607-ad6e72926d9a-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:53.474720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:53.474733    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrd2n\" (UniqueName: \"kubernetes.io/projected/933df945-d492-4035-a607-ad6e72926d9a-kube-api-access-rrd2n\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:54.136598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:54.136555    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"
Oct 30 22:40:54.136598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:54.136557    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g" event={"ID":"933df945-d492-4035-a607-ad6e72926d9a","Type":"ContainerDied","Data":"6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66"}
Oct 30 22:40:54.136816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:54.136635    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6de2ea0ce5d1c6d2fdb64df5574d34da5d7f0b6f8448cac8dedfec5b5ab58a66"
Oct 30 22:40:54.968491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:54.968425    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:55.095220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.095182    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\" (UniqueName: \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\") pod \"b41e0e80-c6b4-4f07-a650-2d0b97692154\" (UID: \"b41e0e80-c6b4-4f07-a650-2d0b97692154\") "
Oct 30 22:40:55.097488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.097451    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb" pod "b41e0e80-c6b4-4f07-a650-2d0b97692154" (UID: "b41e0e80-c6b4-4f07-a650-2d0b97692154"). InnerVolumeSpecName "kube-api-access-sbsfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:40:55.148332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.147366    4882 generic.go:356] "Generic (PLEG): container finished" podID="b41e0e80-c6b4-4f07-a650-2d0b97692154" containerID="29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c" exitCode=137
Oct 30 22:40:55.148332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.147452    4882 scope.go:122] "RemoveContainer" containerID="29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c"
Oct 30 22:40:55.148332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.147657    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:55.163801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.163421    4882 scope.go:122] "RemoveContainer" containerID="29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c"
Oct 30 22:40:55.164042 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:55.164017    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c\": container with ID starting with 29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c not found: ID does not exist" containerID="29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c"
Oct 30 22:40:55.164191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.164164    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c"} err="failed to get container status \"29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c\": rpc error: code = NotFound desc = could not find container \"29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c\": container with ID starting with 29e4e311ced1286dcd48270d8597d1abb39f60788d9d62536493bcd1cc31e40c not found: ID does not exist"
Oct 30 22:40:55.197479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:55.197380    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbsfb\" (UniqueName: \"kubernetes.io/projected/b41e0e80-c6b4-4f07-a650-2d0b97692154-kube-api-access-sbsfb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:40:55.209333 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:55.209288    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-11fc4d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:40:56.944367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:40:56.944336    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" path="/var/lib/kubelet/pods/b41e0e80-c6b4-4f07-a650-2d0b97692154/volumes"
Oct 30 22:40:56.991418 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:40:56.991378    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7658-11fc4d\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" pod="opendatahub/test-curl-7658-11fc4d"
Oct 30 22:41:02.816468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.816431    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:41:02.817281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817263    4882 cpu_manager.go:414] "containerMap: removing container" podUID="933df945-d492-4035-a607-ad6e72926d9a" containerName="revoke-keys"
Oct 30 22:41:02.817334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817284    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="933df945-d492-4035-a607-ad6e72926d9a" containerName="revoke-keys"
Oct 30 22:41:02.817373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817346    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" containerName="test-curl-7658-11fc4d"
Oct 30 22:41:02.817373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817355    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" containerName="test-curl-7658-11fc4d"
Oct 30 22:41:02.817473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817460    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="933df945-d492-4035-a607-ad6e72926d9a" containerName="revoke-keys"
Oct 30 22:41:02.817520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.817485    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b41e0e80-c6b4-4f07-a650-2d0b97692154" containerName="test-curl-7658-11fc4d"
Oct 30 22:41:02.821078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.821056    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:02.878850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.878816    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-8c852b\""
Oct 30 22:41:02.879163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.879125    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-8c852b-dockercfg-mmc9x\""
Oct 30 22:41:02.882080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.882050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:02.882224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.882121    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ch5h\" (UniqueName: \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:02.932533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.932489    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:41:02.947100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.947066    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:41:02.951644 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.951619    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:02.983919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.983884    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:02.984123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.983928    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ch5h\" (UniqueName: \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:02.984123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.984009    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kcknj\" (UniqueName: \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:02.984252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.984192    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:02.985020 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:02.984998    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:03.006322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.006278    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:41:03.046187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.046153    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ch5h\" (UniqueName: \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\") pod \"payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:03.085474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.085439    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:03.085685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.085490    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kcknj\" (UniqueName: \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:03.086424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.086393    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:03.134994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.134959    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:03.148966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.148934    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kcknj\" (UniqueName: \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\") pod \"payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:03.266027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.265986    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:03.509794 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:03.509759    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe4786e8_2760_4354_a1bf_8e5fc9cb8abf.slice/crio-0b897f6e883ab70e8406752194ce5795c409cec93c697307560086bf051edd09 WatchSource:0}: Error finding container 0b897f6e883ab70e8406752194ce5795c409cec93c697307560086bf051edd09: Status 404 returned error can't find the container with id 0b897f6e883ab70e8406752194ce5795c409cec93c697307560086bf051edd09
Oct 30 22:41:03.511986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.511961    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:41:03.651497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:03.651462    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:41:04.213992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.213934    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" event={"ID":"24161188-f356-4958-afdd-c972f05fb851","Type":"ContainerStarted","Data":"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"}
Oct 30 22:41:04.214493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.214018    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" event={"ID":"24161188-f356-4958-afdd-c972f05fb851","Type":"ContainerStarted","Data":"254e93f36e1a881a4b5c74dc36064337cc5ebd264d0745963c9e65db344add39"}
Oct 30 22:41:04.214493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.214123    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:04.215693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.215655    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" event={"ID":"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf","Type":"ContainerStarted","Data":"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"}
Oct 30 22:41:04.215803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.215707    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" event={"ID":"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf","Type":"ContainerStarted","Data":"0b897f6e883ab70e8406752194ce5795c409cec93c697307560086bf051edd09"}
Oct 30 22:41:04.215803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.215747    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:04.334924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.334859    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" podStartSLOduration=2.3348405420000002 podStartE2EDuration="2.334840542s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:04.333210135 +0000 UTC m=+3787.506534562" watchObservedRunningTime="2026-10-30 22:41:04.334840542 +0000 UTC m=+3787.508164970"
Oct 30 22:41:04.455039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:04.454975    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" podStartSLOduration=2.454954336 podStartE2EDuration="2.454954336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:04.453823098 +0000 UTC m=+3787.627147524" watchObservedRunningTime="2026-10-30 22:41:04.454954336 +0000 UTC m=+3787.628278762"
Oct 30 22:41:07.225960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:07.225870    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:41:15.225057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:15.225018    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:41:15.225454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:15.225090    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:41:19.062956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.062916    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7652-bc4f22"]
Oct 30 22:41:19.068079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.068060    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:41:19.121036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.120998    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-bc4f22"]
Oct 30 22:41:19.174271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.174236    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8ssh\" (UniqueName: \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\") pod \"test-curl-7652-bc4f22\" (UID: \"7bd91bca-b1e7-418c-b30a-aa91461d3e10\") " pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:41:19.233490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.233461    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-2jppg\""
Oct 30 22:41:19.275136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.275093    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8ssh\" (UniqueName: \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\") pod \"test-curl-7652-bc4f22\" (UID: \"7bd91bca-b1e7-418c-b30a-aa91461d3e10\") " pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:41:19.336415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.336379    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8ssh\" (UniqueName: \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\") pod \"test-curl-7652-bc4f22\" (UID: \"7bd91bca-b1e7-418c-b30a-aa91461d3e10\") " pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:41:19.387058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.387016    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:41:19.760501 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:19.760469    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7bd91bca_b1e7_418c_b30a_aa91461d3e10.slice/crio-67fbe8baaac951cf34b81022afd9a6276447b1450df461543b851220b0499c2d WatchSource:0}: Error finding container 67fbe8baaac951cf34b81022afd9a6276447b1450df461543b851220b0499c2d: Status 404 returned error can't find the container with id 67fbe8baaac951cf34b81022afd9a6276447b1450df461543b851220b0499c2d
Oct 30 22:41:19.764788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:19.764758    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7652-bc4f22"]
Oct 30 22:41:20.320137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.320088    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-bc4f22" event={"ID":"7bd91bca-b1e7-418c-b30a-aa91461d3e10","Type":"ContainerStarted","Data":"49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439"}
Oct 30 22:41:20.320568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.320146    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7652-bc4f22" event={"ID":"7bd91bca-b1e7-418c-b30a-aa91461d3e10","Type":"ContainerStarted","Data":"67fbe8baaac951cf34b81022afd9a6276447b1450df461543b851220b0499c2d"}
Oct 30 22:41:20.439197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.439139    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7652-bc4f22" podStartSLOduration=1.439120817 podStartE2EDuration="1.439120817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:20.436278011 +0000 UTC m=+3803.609602437" watchObservedRunningTime="2026-10-30 22:41:20.439120817 +0000 UTC m=+3803.612445243"
Oct 30 22:41:20.740376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.740339    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"]
Oct 30 22:41:20.744681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.744658    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.798907 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.798869    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"]
Oct 30 22:41:20.821097 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:41:20.821058    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:41:20.845603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.845561    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:20.845779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.845673    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:41:20.892580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.892548    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pxmk\" (UniqueName: \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.892798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.892619    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.892798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.892766    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.993722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.993646    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pxmk\" (UniqueName: \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.993722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.993687    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.993939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.993779    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.994530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.994502    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:20.996244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:20.996210    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:21.056405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:21.056374    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pxmk\" (UniqueName: \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\") pod \"maas-api-revoke-sub-c61259b9-4pz6q\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:21.059227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:21.059202    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:21.433051 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:21.433021    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f6e8905_af5d_4972_9fcd_6b5dff8af2f2.slice/crio-e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9 WatchSource:0}: Error finding container e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9: Status 404 returned error can't find the container with id e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9
Oct 30 22:41:21.435697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:21.435672    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"]
Oct 30 22:41:22.247753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.247712    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerName="main" containerID="cri-o://fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c" gracePeriod=45
Oct 30 22:41:22.335861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.335829    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q" event={"ID":"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2","Type":"ContainerStarted","Data":"c9329fe8b595deaf3fb5d70684aa01b6d34b3c1e2660ea9f7e4976c8c7e84940"}
Oct 30 22:41:22.336031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.335883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q" event={"ID":"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2","Type":"ContainerStarted","Data":"e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9"}
Oct 30 22:41:22.456629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.456556    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q" podStartSLOduration=2.456540665 podStartE2EDuration="2.456540665s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:22.454161994 +0000 UTC m=+3805.627486423" watchObservedRunningTime="2026-10-30 22:41:22.456540665 +0000 UTC m=+3805.629865130"
Oct 30 22:41:22.517693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.517629    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:41:22.613338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613299    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613340    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613408    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\" (UniqueName: \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613536    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\" (UniqueName: \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613695 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613577    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613695 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613655    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:22.613816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613721    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\") pod \"977372c6-4596-4c33-bedd-98ab46e2aba8\" (UID: \"977372c6-4596-4c33-bedd-98ab46e2aba8\") "
Oct 30 22:41:22.613973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.613911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:22.614112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.614057    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:22.614236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.614211    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:22.614290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.614235    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:22.614290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.614250    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:22.615856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.615835    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:22.616108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.616085    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:22.616108 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.616096    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6" pod "977372c6-4596-4c33-bedd-98ab46e2aba8" (UID: "977372c6-4596-4c33-bedd-98ab46e2aba8"). InnerVolumeSpecName "kube-api-access-78pd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:22.715922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.715884    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/977372c6-4596-4c33-bedd-98ab46e2aba8-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:22.715922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.715918    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-78pd6\" (UniqueName: \"kubernetes.io/projected/977372c6-4596-4c33-bedd-98ab46e2aba8-kube-api-access-78pd6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:22.715922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:22.715928    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/977372c6-4596-4c33-bedd-98ab46e2aba8-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:23.344092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.344055    4882 generic.go:356] "Generic (PLEG): container finished" podID="6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" containerID="c9329fe8b595deaf3fb5d70684aa01b6d34b3c1e2660ea9f7e4976c8c7e84940" exitCode=0
Oct 30 22:41:23.344302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.344160    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q" event={"ID":"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2","Type":"ContainerDied","Data":"c9329fe8b595deaf3fb5d70684aa01b6d34b3c1e2660ea9f7e4976c8c7e84940"}
Oct 30 22:41:23.345994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.345959    4882 generic.go:356] "Generic (PLEG): container finished" podID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerID="fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c" exitCode=0
Oct 30 22:41:23.346197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.346031    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"
Oct 30 22:41:23.346197 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.346170    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" event={"ID":"977372c6-4596-4c33-bedd-98ab46e2aba8","Type":"ContainerDied","Data":"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"}
Oct 30 22:41:23.346292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.346209    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" event={"ID":"977372c6-4596-4c33-bedd-98ab46e2aba8","Type":"ContainerDied","Data":"5fe499fc9954f43de338126c88b157f6c4d60756d92b19400ec24cb7fec15a05"}
Oct 30 22:41:23.346292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.346232    4882 scope.go:122] "RemoveContainer" containerID="fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"
Oct 30 22:41:23.356718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.356697    4882 scope.go:122] "RemoveContainer" containerID="fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"
Oct 30 22:41:23.357031 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:41:23.357005    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c\": container with ID starting with fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c not found: ID does not exist" containerID="fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"
Oct 30 22:41:23.357031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.357044    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c"} err="failed to get container status \"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c\": rpc error: code = NotFound desc = could not find container \"fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c\": container with ID starting with fd9bc824dfbfffb61678160de8d26438604d40816afe477ee3b17aa8878ea82c not found: ID does not exist"
Oct 30 22:41:23.494352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.494306    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerName="main" probeResult="failure" output="Get \"https://10.132.1.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Oct 30 22:41:23.577281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.577235    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:41:23.581452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:23.581419    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-43d592/auth-test-model-43d592-kserve-7f76f46cc7-v5mqw"]
Oct 30 22:41:24.503848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.503821    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:24.637190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.637095    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\") pod \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") "
Oct 30 22:41:24.637358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.637218    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\") pod \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") "
Oct 30 22:41:24.637358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.637248    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\" (UniqueName: \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\") pod \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\" (UID: \"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2\") "
Oct 30 22:41:24.637568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.637535    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca" pod "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" (UID: "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:24.638047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.637993    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:24.640193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.640165    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca" pod "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" (UID: "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:24.640276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.640221    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk" pod "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" (UID: "6f6e8905-af5d-4972-9fcd-6b5dff8af2f2"). InnerVolumeSpecName "kube-api-access-8pxmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:24.738894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.738858    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:24.738894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.738890    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pxmk\" (UniqueName: \"kubernetes.io/projected/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2-kube-api-access-8pxmk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:24.943989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:24.943902    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" path="/var/lib/kubelet/pods/977372c6-4596-4c33-bedd-98ab46e2aba8/volumes"
Oct 30 22:41:25.280926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.280845    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:41:25.281435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281421    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" containerName="revoke-keys"
Oct 30 22:41:25.281480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281437    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" containerName="revoke-keys"
Oct 30 22:41:25.281480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281465    4882 cpu_manager.go:414] "containerMap: removing container" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerName="main"
Oct 30 22:41:25.281480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281471    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerName="main"
Oct 30 22:41:25.281628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281615    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="977372c6-4596-4c33-bedd-98ab46e2aba8" containerName="main"
Oct 30 22:41:25.281705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.281637    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" containerName="revoke-keys"
Oct 30 22:41:25.285483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.285466    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.341463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.341420    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:41:25.364336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.364298    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"
Oct 30 22:41:25.364336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.364315    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q" event={"ID":"6f6e8905-af5d-4972-9fcd-6b5dff8af2f2","Type":"ContainerDied","Data":"e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9"}
Oct 30 22:41:25.364648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.364364    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d2ad2cf7bbe0e8146bfd5e0bb82be1b10c6cbc3b7e3384110b0299061052a9"
Oct 30 22:41:25.386835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.386805    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-2cc983\""
Oct 30 22:41:25.386835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.386806    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-2cc983-dockercfg-d44bs\""
Oct 30 22:41:25.424689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.424653    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:41:25.428812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.428793    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.447690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.447663    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.447856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.447839    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnh8r\" (UniqueName: \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.483774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.483736    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:41:25.549132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.549043    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnh8r\" (UniqueName: \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.549132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.549116    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rdswl\" (UniqueName: \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.549550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.549161    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.549550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.549247    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.549948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.549926    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.611410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.611369    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnh8r\" (UniqueName: \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\") pod \"payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:25.650519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.650474    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rdswl\" (UniqueName: \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.650778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.650555    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.651286 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.651262    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.712882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.712843    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rdswl\" (UniqueName: \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\") pod \"payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.742565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.742529    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:25.898842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:25.898804    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:26.123121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.123082    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:41:26.124429 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:26.124399    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3260293a_41f9_44a5_aaa0_1bc61d2d159f.slice/crio-3d4408ddab6829035f6937475cc949cc3cad9fe9d9c4c72b8da4317fe940c2e4 WatchSource:0}: Error finding container 3d4408ddab6829035f6937475cc949cc3cad9fe9d9c4c72b8da4317fe940c2e4: Status 404 returned error can't find the container with id 3d4408ddab6829035f6937475cc949cc3cad9fe9d9c4c72b8da4317fe940c2e4
Oct 30 22:41:26.275466 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:26.275433    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee18f269_aa08_4805_bbdd_10d7706f1707.slice/crio-b546da6cc1a1fddcdef017c858c457ba689e59e68bc395fb53db1c2e8a030300 WatchSource:0}: Error finding container b546da6cc1a1fddcdef017c858c457ba689e59e68bc395fb53db1c2e8a030300: Status 404 returned error can't find the container with id b546da6cc1a1fddcdef017c858c457ba689e59e68bc395fb53db1c2e8a030300
Oct 30 22:41:26.278190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.278166    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:41:26.372463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.372381    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" event={"ID":"ee18f269-aa08-4805-bbdd-10d7706f1707","Type":"ContainerStarted","Data":"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"}
Oct 30 22:41:26.372463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.372443    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:26.372687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.372470    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" event={"ID":"ee18f269-aa08-4805-bbdd-10d7706f1707","Type":"ContainerStarted","Data":"b546da6cc1a1fddcdef017c858c457ba689e59e68bc395fb53db1c2e8a030300"}
Oct 30 22:41:26.374375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.374350    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" event={"ID":"3260293a-41f9-44a5-aaa0-1bc61d2d159f","Type":"ContainerStarted","Data":"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"}
Oct 30 22:41:26.374469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.374398    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" event={"ID":"3260293a-41f9-44a5-aaa0-1bc61d2d159f","Type":"ContainerStarted","Data":"3d4408ddab6829035f6937475cc949cc3cad9fe9d9c4c72b8da4317fe940c2e4"}
Oct 30 22:41:26.374469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.374420    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:26.493000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.492943    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" podStartSLOduration=1.492927395 podStartE2EDuration="1.492927395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:26.490295767 +0000 UTC m=+3809.663620194" watchObservedRunningTime="2026-10-30 22:41:26.492927395 +0000 UTC m=+3809.666251868"
Oct 30 22:41:26.612498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:26.612431    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" podStartSLOduration=1.6124110379999999 podStartE2EDuration="1.612411038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:26.60829068 +0000 UTC m=+3809.781615108" watchObservedRunningTime="2026-10-30 22:41:26.612411038 +0000 UTC m=+3809.785735468"
Oct 30 22:41:30.190578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:30.190534    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:41:30.191005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:30.190779    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" containerID="cri-o://3fdb9f7be0cf9d404c91ba0fccdf3050103422ebf44ea1197eaf001052f76021" gracePeriod=30
Oct 30 22:41:31.524416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.524382    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"]
Oct 30 22:41:31.528387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.528365    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.582635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.582602    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"]
Oct 30 22:41:31.616657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.616615    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.616852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.616751    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qctvr\" (UniqueName: \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.616852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.616810    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.629415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.629383    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:41:31.629569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.629434    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:41:31.718349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.718316    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qctvr\" (UniqueName: \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.718349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.718353    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.718618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.718453    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.719229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.719201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.720793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.720774    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.780246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.780166    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qctvr\" (UniqueName: \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\") pod \"maas-api-revoke-sub-3f1f2dc2-l56v9\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:31.843360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:31.843322    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:32.215548 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:32.215518    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9220c2f3_658c_47e8_ad87_f1bae051dd19.slice/crio-bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b WatchSource:0}: Error finding container bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b: Status 404 returned error can't find the container with id bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b
Oct 30 22:41:32.218437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:32.218410    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"]
Oct 30 22:41:32.414789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:32.414746    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9" event={"ID":"9220c2f3-658c-47e8-ad87-f1bae051dd19","Type":"ContainerStarted","Data":"bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b"}
Oct 30 22:41:33.422932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:33.422895    4882 generic.go:356] "Generic (PLEG): container finished" podID="9220c2f3-658c-47e8-ad87-f1bae051dd19" containerID="bcfa644d3aae00e5b13d35f6287ea87cdd8a83cd3aa9a81f23f55fc9a54118b5" exitCode=0
Oct 30 22:41:33.423341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:33.422991    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9" event={"ID":"9220c2f3-658c-47e8-ad87-f1bae051dd19","Type":"ContainerDied","Data":"bcfa644d3aae00e5b13d35f6287ea87cdd8a83cd3aa9a81f23f55fc9a54118b5"}
Oct 30 22:41:34.540547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.540507    4882 patch_prober.go:29] interesting pod/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:41:34.541045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.540581    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:41:34.587515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.587486    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:34.655308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.655278    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\" (UniqueName: \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\") pod \"9220c2f3-658c-47e8-ad87-f1bae051dd19\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") "
Oct 30 22:41:34.655472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.655335    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\") pod \"9220c2f3-658c-47e8-ad87-f1bae051dd19\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") "
Oct 30 22:41:34.655472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.655365    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\") pod \"9220c2f3-658c-47e8-ad87-f1bae051dd19\" (UID: \"9220c2f3-658c-47e8-ad87-f1bae051dd19\") "
Oct 30 22:41:34.655792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.655765    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca" pod "9220c2f3-658c-47e8-ad87-f1bae051dd19" (UID: "9220c2f3-658c-47e8-ad87-f1bae051dd19"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:34.655932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.655918    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:34.657548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.657515    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca" pod "9220c2f3-658c-47e8-ad87-f1bae051dd19" (UID: "9220c2f3-658c-47e8-ad87-f1bae051dd19"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:41:34.657674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.657595    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr" pod "9220c2f3-658c-47e8-ad87-f1bae051dd19" (UID: "9220c2f3-658c-47e8-ad87-f1bae051dd19"). InnerVolumeSpecName "kube-api-access-qctvr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:34.756849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.756774    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qctvr\" (UniqueName: \"kubernetes.io/projected/9220c2f3-658c-47e8-ad87-f1bae051dd19-kube-api-access-qctvr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:34.756849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:34.756800    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9220c2f3-658c-47e8-ad87-f1bae051dd19-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.436973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.436937    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"
Oct 30 22:41:35.436973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.436954    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9" event={"ID":"9220c2f3-658c-47e8-ad87-f1bae051dd19","Type":"ContainerDied","Data":"bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b"}
Oct 30 22:41:35.437258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.437003    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bfd498e75e3ef578b78644cae21b77bb29d562924bd0153c838ccaf3638e4d9b"
Oct 30 22:41:35.439101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.439072    4882 generic.go:356] "Generic (PLEG): container finished" podID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerID="3fdb9f7be0cf9d404c91ba0fccdf3050103422ebf44ea1197eaf001052f76021" exitCode=0
Oct 30 22:41:35.439260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.439144    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" event={"ID":"803c8a38-8f24-4050-bc51-09f62fb19ddc","Type":"ContainerDied","Data":"3fdb9f7be0cf9d404c91ba0fccdf3050103422ebf44ea1197eaf001052f76021"}
Oct 30 22:41:35.450476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.450451    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:41:35.567501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567210    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.567501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567283    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.567501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567382    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.567501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567479    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567518    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567543    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567549    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567625    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567661    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567725    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567766    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\") pod \"803c8a38-8f24-4050-bc51-09f62fb19ddc\" (UID: \"803c8a38-8f24-4050-bc51-09f62fb19ddc\") "
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.567882    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568105    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.568256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568208    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:41:35.569206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568373    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.569206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568392    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.569206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568407    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.569206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568422    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.569206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.568436    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/803c8a38-8f24-4050-bc51-09f62fb19ddc-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.570103 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.570079    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:41:35.570305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.570283    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "kube-api-access-tmd6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.570415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.570396    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:41:35.570787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.570767    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy" pod "803c8a38-8f24-4050-bc51-09f62fb19ddc" (UID: "803c8a38-8f24-4050-bc51-09f62fb19ddc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:41:35.670194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.670157    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.670194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.670186    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.670194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.670196    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tmd6v\" (UniqueName: \"kubernetes.io/projected/803c8a38-8f24-4050-bc51-09f62fb19ddc-kube-api-access-tmd6v\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.670461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.670205    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/803c8a38-8f24-4050-bc51-09f62fb19ddc-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:41:35.974927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.974892    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:41:35.975480 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975467    4882 cpu_manager.go:414] "containerMap: removing container" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy"
Oct 30 22:41:35.975535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975481    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy"
Oct 30 22:41:35.975535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975513    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9220c2f3-658c-47e8-ad87-f1bae051dd19" containerName="revoke-keys"
Oct 30 22:41:35.975535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975518    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9220c2f3-658c-47e8-ad87-f1bae051dd19" containerName="revoke-keys"
Oct 30 22:41:35.975668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975632    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" containerName="istio-proxy"
Oct 30 22:41:35.975668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.975654    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9220c2f3-658c-47e8-ad87-f1bae051dd19" containerName="revoke-keys"
Oct 30 22:41:35.979479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:35.979455    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.032649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.032615    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-1a5f4a\""
Oct 30 22:41:36.032854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.032750    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-1a5f4a-dockercfg-fqlxq\""
Oct 30 22:41:36.034786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.034756    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:41:36.074569 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.074532    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.074779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.074755    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pn84l\" (UniqueName: \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.162430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.162393    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:41:36.166775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.166750    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.175828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.175800    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pn84l\" (UniqueName: \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.175990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.175887    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.176637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.176619    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.221968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.221921    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:41:36.238002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.237908    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pn84l\" (UniqueName: \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\") pod \"payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.277405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.277357    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.277630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.277426    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzmzb\" (UniqueName: \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.293238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.293204    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:36.378664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.378624    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.378881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.378679    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzmzb\" (UniqueName: \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.379462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.379435    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.441238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.441201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzmzb\" (UniqueName: \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\") pod \"payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.446902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.446874    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"
Oct 30 22:41:36.446902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.446894    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf" event={"ID":"803c8a38-8f24-4050-bc51-09f62fb19ddc","Type":"ContainerDied","Data":"caebb401719a79f965e84b586c6a3520d43e6cfde4496494fd3e4511f5eade69"}
Oct 30 22:41:36.447114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.446959    4882 scope.go:122] "RemoveContainer" containerID="3fdb9f7be0cf9d404c91ba0fccdf3050103422ebf44ea1197eaf001052f76021"
Oct 30 22:41:36.482314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.482277    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:36.615571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.615540    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:41:36.619655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.619628    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-3c4802-openshift-default-865cb5887f-lkshf"]
Oct 30 22:41:36.719261 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:36.719213    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0755c3c_e2a0_44e1_af29_df19118a6ef4.slice/crio-efb88ef11685ad0754cd865a830178f40185a566766852d2ed4f1d59ba5388c4 WatchSource:0}: Error finding container efb88ef11685ad0754cd865a830178f40185a566766852d2ed4f1d59ba5388c4: Status 404 returned error can't find the container with id efb88ef11685ad0754cd865a830178f40185a566766852d2ed4f1d59ba5388c4
Oct 30 22:41:36.722723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.722697    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:41:36.863671 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:41:36.863634    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode51de84d_55c4_43ab_bcfb_9709d87e7753.slice/crio-cb7224f3a6201d1aaaf00da3bff3ee197154bc21d5ad9d43b6a0f4d5d379e752 WatchSource:0}: Error finding container cb7224f3a6201d1aaaf00da3bff3ee197154bc21d5ad9d43b6a0f4d5d379e752: Status 404 returned error can't find the container with id cb7224f3a6201d1aaaf00da3bff3ee197154bc21d5ad9d43b6a0f4d5d379e752
Oct 30 22:41:36.866830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.866801    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:41:36.949616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:36.948449    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="803c8a38-8f24-4050-bc51-09f62fb19ddc" path="/var/lib/kubelet/pods/803c8a38-8f24-4050-bc51-09f62fb19ddc/volumes"
Oct 30 22:41:37.384640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.384610    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:41:37.384813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.384690    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:41:37.456213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.456144    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" event={"ID":"c0755c3c-e2a0-44e1-af29-df19118a6ef4","Type":"ContainerStarted","Data":"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"}
Oct 30 22:41:37.456401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.456236    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:37.456401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.456261    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" event={"ID":"c0755c3c-e2a0-44e1-af29-df19118a6ef4","Type":"ContainerStarted","Data":"efb88ef11685ad0754cd865a830178f40185a566766852d2ed4f1d59ba5388c4"}
Oct 30 22:41:37.457874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.457850    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" event={"ID":"e51de84d-55c4-43ab-bcfb-9709d87e7753","Type":"ContainerStarted","Data":"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"}
Oct 30 22:41:37.458032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.457883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" event={"ID":"e51de84d-55c4-43ab-bcfb-9709d87e7753","Type":"ContainerStarted","Data":"cb7224f3a6201d1aaaf00da3bff3ee197154bc21d5ad9d43b6a0f4d5d379e752"}
Oct 30 22:41:37.458032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.457935    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:37.742165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.742044    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" podStartSLOduration=2.742022742 podStartE2EDuration="2.742022742s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:37.738876113 +0000 UTC m=+3820.912200540" watchObservedRunningTime="2026-10-30 22:41:37.742022742 +0000 UTC m=+3820.915347262"
Oct 30 22:41:37.857715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:37.857660    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" podStartSLOduration=1.857644447 podStartE2EDuration="1.857644447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:41:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:41:37.85545251 +0000 UTC m=+3821.028776938" watchObservedRunningTime="2026-10-30 22:41:37.857644447 +0000 UTC m=+3821.030968875"
Oct 30 22:41:48.469186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:48.469150    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:41:48.469830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:48.469230    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:41:49.002920 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:41:49.002877    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:41:50.809315 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:41:50.809274    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:42:00.057010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:00.056975    4882 scope.go:122] "RemoveContainer" containerID="3de6d1ebb041b5abcb85bb71569e0115e3a544558da6f98173c921e98c30b6bd"
Oct 30 22:42:20.809517 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:20.809473    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:42:21.256259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.256221    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7652-bc4f22"]
Oct 30 22:42:21.258527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.256445    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7652-bc4f22" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" containerName="test-curl-7652-bc4f22" containerID="cri-o://49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439" gracePeriod=2
Oct 30 22:42:21.260061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.260021    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7652-bc4f22"]
Oct 30 22:42:21.308815 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:21.308768    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-bc4f22\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:21.579287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.579184    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"]
Oct 30 22:42:21.580183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.580165    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" containerName="test-curl-7652-bc4f22"
Oct 30 22:42:21.580240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.580185    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" containerName="test-curl-7652-bc4f22"
Oct 30 22:42:21.580328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.580316    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" containerName="test-curl-7652-bc4f22"
Oct 30 22:42:21.584541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.584514    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.606346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.606292    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqc8k\" (UniqueName: \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.606540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.606357    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.606540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.606518    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.635210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.635171    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"]
Oct 30 22:42:21.636532 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:21.636492    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-bc4f22\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:21.638571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.638543    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:21.681928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.681896    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:42:21.707269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.707224    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.707447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.707317    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqc8k\" (UniqueName: \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.707447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.707357    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.708068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.708042    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.709763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.709741    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.771609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.771555    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqc8k\" (UniqueName: \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\") pod \"maas-api-revoke-sub-c44ede00-7qzkw\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:21.901069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:21.901027    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:22.285708 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:42:22.285671    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod951e3eec_6db5_49eb_a7ad_8df628ddbb25.slice/crio-b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1 WatchSource:0}: Error finding container b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1: Status 404 returned error can't find the container with id b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1
Oct 30 22:42:22.290363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:22.290264    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"]
Oct 30 22:42:22.759538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:22.759500    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw" event={"ID":"951e3eec-6db5-49eb-a7ad-8df628ddbb25","Type":"ContainerStarted","Data":"f5e2268be0d503240fcedef0b66a8477050ae175332ef8472b4e2c18ef61f1fb"}
Oct 30 22:42:22.759793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:22.759614    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw" event={"ID":"951e3eec-6db5-49eb-a7ad-8df628ddbb25","Type":"ContainerStarted","Data":"b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1"}
Oct 30 22:42:22.811685 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:22.811596    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-bc4f22\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:22.929754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:22.929699    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw" podStartSLOduration=1.9296857699999999 podStartE2EDuration="1.92968577s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:22.928568078 +0000 UTC m=+3866.101892506" watchObservedRunningTime="2026-10-30 22:42:22.92968577 +0000 UTC m=+3866.103010196"
Oct 30 22:42:23.498716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.498684    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:23.528945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.528906    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\" (UniqueName: \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\") pod \"7bd91bca-b1e7-418c-b30a-aa91461d3e10\" (UID: \"7bd91bca-b1e7-418c-b30a-aa91461d3e10\") "
Oct 30 22:42:23.531115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.531085    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh" pod "7bd91bca-b1e7-418c-b30a-aa91461d3e10" (UID: "7bd91bca-b1e7-418c-b30a-aa91461d3e10"). InnerVolumeSpecName "kube-api-access-j8ssh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:23.630464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.630419    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8ssh\" (UniqueName: \"kubernetes.io/projected/7bd91bca-b1e7-418c-b30a-aa91461d3e10-kube-api-access-j8ssh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:23.768370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.768272    4882 generic.go:356] "Generic (PLEG): container finished" podID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" containerID="49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439" exitCode=137
Oct 30 22:42:23.768370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.768316    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:23.768370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.768364    4882 scope.go:122] "RemoveContainer" containerID="49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439"
Oct 30 22:42:23.770359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.770333    4882 generic.go:356] "Generic (PLEG): container finished" podID="951e3eec-6db5-49eb-a7ad-8df628ddbb25" containerID="f5e2268be0d503240fcedef0b66a8477050ae175332ef8472b4e2c18ef61f1fb" exitCode=0
Oct 30 22:42:23.770504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.770399    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw" event={"ID":"951e3eec-6db5-49eb-a7ad-8df628ddbb25","Type":"ContainerDied","Data":"f5e2268be0d503240fcedef0b66a8477050ae175332ef8472b4e2c18ef61f1fb"}
Oct 30 22:42:23.778692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.778673    4882 scope.go:122] "RemoveContainer" containerID="49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439"
Oct 30 22:42:23.779023 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:23.779000    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439\": container with ID starting with 49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439 not found: ID does not exist" containerID="49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439"
Oct 30 22:42:23.779104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:23.779033    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439"} err="failed to get container status \"49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439\": rpc error: code = NotFound desc = could not find container \"49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439\": container with ID starting with 49ec612e2d1045adba400fa874537c379258c803e4e1fc7f037c1984bd1ce439 not found: ID does not exist"
Oct 30 22:42:23.944551 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:23.944514    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-bc4f22\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:24.937713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:24.937681    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:24.943255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:24.943222    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" path="/var/lib/kubelet/pods/7bd91bca-b1e7-418c-b30a-aa91461d3e10/volumes"
Oct 30 22:42:24.989679 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:24.989636    4882 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7652-bc4f22\" is forbidden: User \"system:node:ip-10-0-141-82.us-west-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-82.us-west-2.compute.internal' and this object" podUID="7bd91bca-b1e7-418c-b30a-aa91461d3e10" pod="opendatahub/test-curl-7652-bc4f22"
Oct 30 22:42:25.047570 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.047481    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\") pod \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") "
Oct 30 22:42:25.047753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.047609    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\") pod \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") "
Oct 30 22:42:25.047753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.047700    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\" (UniqueName: \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\") pod \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\" (UID: \"951e3eec-6db5-49eb-a7ad-8df628ddbb25\") "
Oct 30 22:42:25.048012 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.047988    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca" pod "951e3eec-6db5-49eb-a7ad-8df628ddbb25" (UID: "951e3eec-6db5-49eb-a7ad-8df628ddbb25"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:25.048153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.048139    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:25.049663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.049634    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca" pod "951e3eec-6db5-49eb-a7ad-8df628ddbb25" (UID: "951e3eec-6db5-49eb-a7ad-8df628ddbb25"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:25.049779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.049671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k" pod "951e3eec-6db5-49eb-a7ad-8df628ddbb25" (UID: "951e3eec-6db5-49eb-a7ad-8df628ddbb25"). InnerVolumeSpecName "kube-api-access-bqc8k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:25.149007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.148974    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/951e3eec-6db5-49eb-a7ad-8df628ddbb25-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:25.149007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.149005    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqc8k\" (UniqueName: \"kubernetes.io/projected/951e3eec-6db5-49eb-a7ad-8df628ddbb25-kube-api-access-bqc8k\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:25.785778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.785748    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"
Oct 30 22:42:25.785778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.785773    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw" event={"ID":"951e3eec-6db5-49eb-a7ad-8df628ddbb25","Type":"ContainerDied","Data":"b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1"}
Oct 30 22:42:25.786016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:25.785817    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b5e78c41fb3cf885593296d3eaac9d2ccfbb0a45d683d742ad7d6e3dd5b247a1"
Oct 30 22:42:32.313801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.313765    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"]
Oct 30 22:42:32.314391 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.314375    4882 cpu_manager.go:414] "containerMap: removing container" podUID="951e3eec-6db5-49eb-a7ad-8df628ddbb25" containerName="revoke-keys"
Oct 30 22:42:32.314443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.314392    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="951e3eec-6db5-49eb-a7ad-8df628ddbb25" containerName="revoke-keys"
Oct 30 22:42:32.314514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.314504    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="951e3eec-6db5-49eb-a7ad-8df628ddbb25" containerName="revoke-keys"
Oct 30 22:42:32.318066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.318044    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.372854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.372819    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"]
Oct 30 22:42:32.418766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.418729    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:42:32.418950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.418770    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:42:32.429889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.429854    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m8zvc\" (UniqueName: \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.430080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.430055    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.430128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.430113    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.531608 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.531563    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.531817 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.531678    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m8zvc\" (UniqueName: \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.531896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.531853    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.532528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.532492    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.534058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.534035    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.593709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.593673    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m8zvc\" (UniqueName: \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\") pod \"maas-api-revoke-sub-0ac779d3-hbs6q\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:32.631854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:32.631812    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:33.011562 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:42:33.011500    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1655dbfa_f71b_470a_9011_5038a14007f9.slice/crio-5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d WatchSource:0}: Error finding container 5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d: Status 404 returned error can't find the container with id 5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d
Oct 30 22:42:33.015088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:33.015060    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"]
Oct 30 22:42:33.845062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:33.845025    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q" event={"ID":"1655dbfa-f71b-470a-9011-5038a14007f9","Type":"ContainerStarted","Data":"f9f6e18abf510ac7bf90b9aa137fd15d48b7bc12082fd53419308970dd73279c"}
Oct 30 22:42:33.845549 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:33.845114    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q" event={"ID":"1655dbfa-f71b-470a-9011-5038a14007f9","Type":"ContainerStarted","Data":"5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d"}
Oct 30 22:42:33.968276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:33.968223    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q" podStartSLOduration=1.96820739 podStartE2EDuration="1.96820739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:42:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:42:33.964397004 +0000 UTC m=+3877.137721430" watchObservedRunningTime="2026-10-30 22:42:33.96820739 +0000 UTC m=+3877.141531818"
Oct 30 22:42:34.852938 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:34.852898    4882 generic.go:356] "Generic (PLEG): container finished" podID="1655dbfa-f71b-470a-9011-5038a14007f9" containerID="f9f6e18abf510ac7bf90b9aa137fd15d48b7bc12082fd53419308970dd73279c" exitCode=0
Oct 30 22:42:34.853363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:34.852964    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q" event={"ID":"1655dbfa-f71b-470a-9011-5038a14007f9","Type":"ContainerDied","Data":"f9f6e18abf510ac7bf90b9aa137fd15d48b7bc12082fd53419308970dd73279c"}
Oct 30 22:42:36.032386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.032356    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:36.174770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.174688    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\" (UniqueName: \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\") pod \"1655dbfa-f71b-470a-9011-5038a14007f9\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") "
Oct 30 22:42:36.174939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.174805    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\") pod \"1655dbfa-f71b-470a-9011-5038a14007f9\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") "
Oct 30 22:42:36.174939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.174869    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\") pod \"1655dbfa-f71b-470a-9011-5038a14007f9\" (UID: \"1655dbfa-f71b-470a-9011-5038a14007f9\") "
Oct 30 22:42:36.175259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.175226    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca" pod "1655dbfa-f71b-470a-9011-5038a14007f9" (UID: "1655dbfa-f71b-470a-9011-5038a14007f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:42:36.175465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.175444    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:36.176969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.176941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca" pod "1655dbfa-f71b-470a-9011-5038a14007f9" (UID: "1655dbfa-f71b-470a-9011-5038a14007f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:42:36.177070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.176991    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc" pod "1655dbfa-f71b-470a-9011-5038a14007f9" (UID: "1655dbfa-f71b-470a-9011-5038a14007f9"). InnerVolumeSpecName "kube-api-access-m8zvc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:42:36.276954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.276912    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m8zvc\" (UniqueName: \"kubernetes.io/projected/1655dbfa-f71b-470a-9011-5038a14007f9-kube-api-access-m8zvc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:36.276954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.276947    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1655dbfa-f71b-470a-9011-5038a14007f9-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:42:36.868255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.868178    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q" event={"ID":"1655dbfa-f71b-470a-9011-5038a14007f9","Type":"ContainerDied","Data":"5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d"}
Oct 30 22:42:36.868255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.868226    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5a019ea0a0697d2dbd0ba11aaaa6c4348f361fa00a4a5fa26677b9cf77792d5d"
Oct 30 22:42:36.868255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:36.868189    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"
Oct 30 22:42:50.809134 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:42:50.809083    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:42:50.861292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:50.861256    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:42:57.536712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:57.536683    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:42:57.557729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:42:57.557698    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:43:02.168827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.168784    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:43:02.169466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.169446    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1655dbfa-f71b-470a-9011-5038a14007f9" containerName="revoke-keys"
Oct 30 22:43:02.169512 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.169468    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1655dbfa-f71b-470a-9011-5038a14007f9" containerName="revoke-keys"
Oct 30 22:43:02.169615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.169574    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1655dbfa-f71b-470a-9011-5038a14007f9" containerName="revoke-keys"
Oct 30 22:43:02.173055 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.173038    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.274540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.274503    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.274778 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.274577    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhpwb\" (UniqueName: \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.282098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.282060    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-be6ecc\""
Oct 30 22:43:02.282246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.282100    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-be6ecc-dockercfg-2jhwb\""
Oct 30 22:43:02.287481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.287455    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:43:02.335929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.335888    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:43:02.340245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.340222    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.374574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.374534    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:43:02.375901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.375869    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.376061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.375945    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhpwb\" (UniqueName: \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.376136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.376108    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pw7w8\" (UniqueName: \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.376269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.376251    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.377236 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.377214    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.441235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.441169    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhpwb\" (UniqueName: \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\") pod \"payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.477285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.477239    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pw7w8\" (UniqueName: \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.477485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.477430    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.478225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.478198    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.486849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.486822    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:02.539954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.539910    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pw7w8\" (UniqueName: \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\") pod \"payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.666699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.666659    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:02.876643 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:02.876609    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa067076_f410_4aeb_90de_0251ba09a294.slice/crio-8d436c78b3a73ea36f9ba7963aff3cb7e1cc8b2e7d4ec3db3b7529c7ca0be094 WatchSource:0}: Error finding container 8d436c78b3a73ea36f9ba7963aff3cb7e1cc8b2e7d4ec3db3b7529c7ca0be094: Status 404 returned error can't find the container with id 8d436c78b3a73ea36f9ba7963aff3cb7e1cc8b2e7d4ec3db3b7529c7ca0be094
Oct 30 22:43:02.879097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:02.879076    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:43:03.042751 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:03.042713    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod165fd979_8b88_482e_89a4_c27e621f1a42.slice/crio-461831d9dc1c3e5c2cbbb4959d1a6e05d9534101bca94d92f813d94f6f3d22f2 WatchSource:0}: Error finding container 461831d9dc1c3e5c2cbbb4959d1a6e05d9534101bca94d92f813d94f6f3d22f2: Status 404 returned error can't find the container with id 461831d9dc1c3e5c2cbbb4959d1a6e05d9534101bca94d92f813d94f6f3d22f2
Oct 30 22:43:03.044724 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:03.044689    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" event={"ID":"aa067076-f410-4aeb-90de-0251ba09a294","Type":"ContainerStarted","Data":"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"}
Oct 30 22:43:03.044862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:03.044744    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" event={"ID":"aa067076-f410-4aeb-90de-0251ba09a294","Type":"ContainerStarted","Data":"8d436c78b3a73ea36f9ba7963aff3cb7e1cc8b2e7d4ec3db3b7529c7ca0be094"}
Oct 30 22:43:03.044862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:03.044802    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:03.045750 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:03.045626    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:43:03.164739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:03.164686    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" podStartSLOduration=1.1646691630000001 podStartE2EDuration="1.164669163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:03.161505318 +0000 UTC m=+3906.334829745" watchObservedRunningTime="2026-10-30 22:43:03.164669163 +0000 UTC m=+3906.337993590"
Oct 30 22:43:04.053927 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:04.053871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" event={"ID":"165fd979-8b88-482e-89a4-c27e621f1a42","Type":"ContainerStarted","Data":"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"}
Oct 30 22:43:04.054447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:04.053975    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:04.054447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:04.054030    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" event={"ID":"165fd979-8b88-482e-89a4-c27e621f1a42","Type":"ContainerStarted","Data":"461831d9dc1c3e5c2cbbb4959d1a6e05d9534101bca94d92f813d94f6f3d22f2"}
Oct 30 22:43:04.172297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:04.172242    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" podStartSLOduration=2.172224689 podStartE2EDuration="2.172224689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:04.17048814 +0000 UTC m=+3907.343812568" watchObservedRunningTime="2026-10-30 22:43:04.172224689 +0000 UTC m=+3907.345549121"
Oct 30 22:43:14.057112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:14.057082    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:43:15.064352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:15.064322    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:43:20.808499 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:43:20.808464    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:43:26.578378 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:26.578339    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:43:41.599989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.599924    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" containerName="main" containerID="cri-o://ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a" gracePeriod=45
Oct 30 22:43:41.604024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.603984    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" containerName="main" probeResult="failure" output="Get \"https://10.132.1.27:8000/ready\": EOF"
Oct 30 22:43:41.869819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.869752    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:43:41.938626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938575    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.938814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938642    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.938814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938682    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.938814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938712    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.938814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938755    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\" (UniqueName: \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.938814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938803    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\" (UniqueName: \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\") pod \"da425850-6851-4880-a556-8a7a2bf61bc1\" (UID: \"da425850-6851-4880-a556-8a7a2bf61bc1\") "
Oct 30 22:43:41.939109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938880    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:41.939109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.938927    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:41.939109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.939000    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:41.939294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.939264    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:41.939294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.939284    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:41.939607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.939297    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:41.941001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.940974    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:43:41.941074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.941035    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "kube-api-access-grpgc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:43:41.941128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:41.941082    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm" pod "da425850-6851-4880-a556-8a7a2bf61bc1" (UID: "da425850-6851-4880-a556-8a7a2bf61bc1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:43:42.040564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.040523    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/da425850-6851-4880-a556-8a7a2bf61bc1-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:42.040564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.040554    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/da425850-6851-4880-a556-8a7a2bf61bc1-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:42.040564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.040567    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grpgc\" (UniqueName: \"kubernetes.io/projected/da425850-6851-4880-a556-8a7a2bf61bc1-kube-api-access-grpgc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:43:42.301694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.301567    4882 generic.go:356] "Generic (PLEG): container finished" podID="da425850-6851-4880-a556-8a7a2bf61bc1" containerID="ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a" exitCode=0
Oct 30 22:43:42.301694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.301646    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"
Oct 30 22:43:42.301694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.301651    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" event={"ID":"da425850-6851-4880-a556-8a7a2bf61bc1","Type":"ContainerDied","Data":"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"}
Oct 30 22:43:42.302000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.301706    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx" event={"ID":"da425850-6851-4880-a556-8a7a2bf61bc1","Type":"ContainerDied","Data":"dee96a006df83d79153fe35f3b87061ad5a180ee83c62822eb1115f45bbcbf3f"}
Oct 30 22:43:42.302000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.301724    4882 scope.go:122] "RemoveContainer" containerID="ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"
Oct 30 22:43:42.313054 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.313019    4882 scope.go:122] "RemoveContainer" containerID="ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"
Oct 30 22:43:42.313355 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:43:42.313334    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a\": container with ID starting with ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a not found: ID does not exist" containerID="ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"
Oct 30 22:43:42.313411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.313365    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a"} err="failed to get container status \"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a\": rpc error: code = NotFound desc = could not find container \"ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a\": container with ID starting with ac09a2908f1ea84dfc03f21dd84f4a9df69c5e2c1ae412bd137a8d80dbe5176a not found: ID does not exist"
Oct 30 22:43:42.472080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.472040    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:43:42.476893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.476862    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w0-6fc60c/auth-test-model-6fc60c-kserve-65cb5fdcc7-zz2qx"]
Oct 30 22:43:42.943753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:42.943720    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" path="/var/lib/kubelet/pods/da425850-6851-4880-a556-8a7a2bf61bc1/volumes"
Oct 30 22:43:44.685956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.685920    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:43:44.686890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.686865    4882 cpu_manager.go:414] "containerMap: removing container" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" containerName="main"
Oct 30 22:43:44.686967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.686893    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" containerName="main"
Oct 30 22:43:44.687163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.687141    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="da425850-6851-4880-a556-8a7a2bf61bc1" containerName="main"
Oct 30 22:43:44.691744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.691719    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.749233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.749174    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Oct 30 22:43:44.755033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.754996    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:43:44.773544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.773506    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.773760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.773681    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.773824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.773753    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.773824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.773794    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.773991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.773972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.774066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.774012    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trzsk\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.774178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.774155    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.774240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.774213    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.774295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.774274    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.799833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.799801    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-b-7a134d-openshift-default-dockercfg-n8xtg\""
Oct 30 22:43:44.875213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875159    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875206    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trzsk\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875254    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875277    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875313    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875347    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875423    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875453    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875483    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875745    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.875918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.875857    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.876073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.876044    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.876212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.876102    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.876269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.876237    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.878276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.878253    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.878439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.878417    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.937755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.937682    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:44.937971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:44.937947    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-trzsk\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\") pod \"e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") " pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:45.007915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:45.007876    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:45.446796 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:45.446760    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54398249_098e_432e_b1bf_2ae76b58d0ad.slice/crio-d21f8916e83212073876c1613bb1546f5c046fd053e58af2fdd7ee2fb9976edc WatchSource:0}: Error finding container d21f8916e83212073876c1613bb1546f5c046fd053e58af2fdd7ee2fb9976edc: Status 404 returned error can't find the container with id d21f8916e83212073876c1613bb1546f5c046fd053e58af2fdd7ee2fb9976edc
Oct 30 22:43:45.450308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:45.450273    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:43:45.450449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:45.450344    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:43:45.450449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:45.450367    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:43:45.450449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:45.450374    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:43:46.330887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:46.330852    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerStarted","Data":"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"}
Oct 30 22:43:46.331309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:46.330908    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerStarted","Data":"d21f8916e83212073876c1613bb1546f5c046fd053e58af2fdd7ee2fb9976edc"}
Oct 30 22:43:46.455338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:46.455281    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podStartSLOduration=2.455266971 podStartE2EDuration="2.455266971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:46.451679302 +0000 UTC m=+3949.625003727" watchObservedRunningTime="2026-10-30 22:43:46.455266971 +0000 UTC m=+3949.628591396"
Oct 30 22:43:47.008158 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:47.008123    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:47.013933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:47.013885    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:47.339296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:47.339262    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:47.340248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:47.340224    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:43:47.910406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:47.910369    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:43:50.809640 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:43:50.809599    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:43:52.812819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.812786    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:43:52.817124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.817102    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:52.871706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.871672    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:43:52.917880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.917837    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-7a134d\""
Oct 30 22:43:52.917880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.917838    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-7a134d-dockercfg-4lvdx\""
Oct 30 22:43:52.959459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.959427    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:43:52.963803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.963783    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:52.971191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.971126    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j825p\" (UniqueName: \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:52.971337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:52.971309    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.018222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.018188    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:43:53.072931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.072834    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j825p\" (UniqueName: \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.073114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.072930    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.073114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.072978    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.073252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.073134    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr7g6\" (UniqueName: \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.073787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.073766    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.135233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.135196    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j825p\" (UniqueName: \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\") pod \"payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.174537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.174489    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr7g6\" (UniqueName: \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.174773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.174722    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.175433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.175407    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.238253 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.238215    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr7g6\" (UniqueName: \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\") pod \"payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.278305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.278263    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:53.431842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.431804    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:53.658361 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:53.658327    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c77ac22_68a0_46a7_b7f6_3e378e9874df.slice/crio-9273a5fca265e214549ab84b36215ca4ab4785024157dff5fc6d74d7cb63a059 WatchSource:0}: Error finding container 9273a5fca265e214549ab84b36215ca4ab4785024157dff5fc6d74d7cb63a059: Status 404 returned error can't find the container with id 9273a5fca265e214549ab84b36215ca4ab4785024157dff5fc6d74d7cb63a059
Oct 30 22:43:53.661096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.661073    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:43:53.804705 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:53.804664    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b0aed04_7985_4bd7_8602_ad9a1fdb8644.slice/crio-20df4cab038cec870ee8707adfc4e1d3375317f9ce4ac656fb3bc5a37366e9b9 WatchSource:0}: Error finding container 20df4cab038cec870ee8707adfc4e1d3375317f9ce4ac656fb3bc5a37366e9b9: Status 404 returned error can't find the container with id 20df4cab038cec870ee8707adfc4e1d3375317f9ce4ac656fb3bc5a37366e9b9
Oct 30 22:43:53.808110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:53.808085    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:43:54.397668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.397627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" event={"ID":"3c77ac22-68a0-46a7-b7f6-3e378e9874df","Type":"ContainerStarted","Data":"1a72771695a05af9b29cf3391cfc2fb727325e8c502586e4b47a9857faca2cd5"}
Oct 30 22:43:54.398221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.397701    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:43:54.398221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.397723    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" event={"ID":"3c77ac22-68a0-46a7-b7f6-3e378e9874df","Type":"ContainerStarted","Data":"9273a5fca265e214549ab84b36215ca4ab4785024157dff5fc6d74d7cb63a059"}
Oct 30 22:43:54.399679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.399634    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" event={"ID":"9b0aed04-7985-4bd7-8602-ad9a1fdb8644","Type":"ContainerStarted","Data":"232d55f2aa3107cd13463fd49191c33296059150c9534cfed8c047cd9750d4b1"}
Oct 30 22:43:54.399816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.399694    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" event={"ID":"9b0aed04-7985-4bd7-8602-ad9a1fdb8644","Type":"ContainerStarted","Data":"20df4cab038cec870ee8707adfc4e1d3375317f9ce4ac656fb3bc5a37366e9b9"}
Oct 30 22:43:54.399816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.399715    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:43:54.535357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.535303    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" podStartSLOduration=2.53528743 podStartE2EDuration="2.53528743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:54.531772075 +0000 UTC m=+3957.705096501" watchObservedRunningTime="2026-10-30 22:43:54.53528743 +0000 UTC m=+3957.708611850"
Oct 30 22:43:54.653432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:54.653318    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" podStartSLOduration=2.653303412 podStartE2EDuration="2.653303412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:54.651051771 +0000 UTC m=+3957.824376198" watchObservedRunningTime="2026-10-30 22:43:54.653303412 +0000 UTC m=+3957.826627837"
Oct 30 22:43:57.873735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.873698    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:43:57.878269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.878247    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:57.919689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.919645    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:57.919899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.919702    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmpvq\" (UniqueName: \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:57.935876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.935797    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-5a5534-dockercfg-m2mhg\""
Oct 30 22:43:57.986105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.986026    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-5a5534\""
Oct 30 22:43:57.990173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:57.990140    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:43:58.021321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.021286    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:58.021321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.021333    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmpvq\" (UniqueName: \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:58.022138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.022109    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:58.030073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.030043    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:43:58.034464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.034445    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.084506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.084471    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmpvq\" (UniqueName: \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\") pod \"payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:58.088561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.088534    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:43:58.122517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.122487    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjdtd\" (UniqueName: \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.122773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.122744    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.192410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.192321    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:58.223779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.223745    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.223964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.223820    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjdtd\" (UniqueName: \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.224473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.224451    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.286143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.286103    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjdtd\" (UniqueName: \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\") pod \"payload-processing-e2e-autores-5a5534-7ffd856679-24hkw\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") " pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.349348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.349316    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:58.568054 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:58.568018    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25af3c2_a039_44d4_a197_1a726eefb161.slice/crio-ef14e718c24a7b28c7cd3435d144f7713f9eae1f4e8b1493856115defb3f9f08 WatchSource:0}: Error finding container ef14e718c24a7b28c7cd3435d144f7713f9eae1f4e8b1493856115defb3f9f08: Status 404 returned error can't find the container with id ef14e718c24a7b28c7cd3435d144f7713f9eae1f4e8b1493856115defb3f9f08
Oct 30 22:43:58.572060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.572029    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:43:58.727406 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:43:58.727366    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod480b7a25_cb3c_44b9_a04f_861d1f7cd91a.slice/crio-201d09a948b715fa2684d91127ced4c358cc071772dfd63172eae8d1189aa996 WatchSource:0}: Error finding container 201d09a948b715fa2684d91127ced4c358cc071772dfd63172eae8d1189aa996: Status 404 returned error can't find the container with id 201d09a948b715fa2684d91127ced4c358cc071772dfd63172eae8d1189aa996
Oct 30 22:43:58.731048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:58.730958    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:43:59.437196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.437157    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" event={"ID":"480b7a25-cb3c-44b9-a04f-861d1f7cd91a","Type":"ContainerStarted","Data":"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"}
Oct 30 22:43:59.437732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.437211    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" event={"ID":"480b7a25-cb3c-44b9-a04f-861d1f7cd91a","Type":"ContainerStarted","Data":"201d09a948b715fa2684d91127ced4c358cc071772dfd63172eae8d1189aa996"}
Oct 30 22:43:59.437732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.437242    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:43:59.438955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.438929    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" event={"ID":"d25af3c2-a039-44d4-a197-1a726eefb161","Type":"ContainerStarted","Data":"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"}
Oct 30 22:43:59.439064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.438973    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" event={"ID":"d25af3c2-a039-44d4-a197-1a726eefb161","Type":"ContainerStarted","Data":"ef14e718c24a7b28c7cd3435d144f7713f9eae1f4e8b1493856115defb3f9f08"}
Oct 30 22:43:59.439064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.438993    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:43:59.558680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.558614    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" podStartSLOduration=2.558580411 podStartE2EDuration="2.558580411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:59.55339905 +0000 UTC m=+3962.726723478" watchObservedRunningTime="2026-10-30 22:43:59.558580411 +0000 UTC m=+3962.731904839"
Oct 30 22:43:59.673330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:43:59.673281    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" podStartSLOduration=2.673266902 podStartE2EDuration="2.673266902s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:43:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:43:59.670143053 +0000 UTC m=+3962.843467479" watchObservedRunningTime="2026-10-30 22:43:59.673266902 +0000 UTC m=+3962.846591356"
Oct 30 22:44:05.409285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:05.409251    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:44:05.409749 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:05.409335    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:44:10.449166 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:10.449082    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:44:10.449646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:10.449180    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:44:18.231808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.231767    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:44:18.236149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.236128    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.291925 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.291880    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:44:18.337248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.337213    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-6e0b99-dockercfg-gcqh9\""
Oct 30 22:44:18.337248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.337213    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-6e0b99\""
Oct 30 22:44:18.366138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.366099    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9tms\" (UniqueName: \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.366358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.366222    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.384725 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.384685    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:44:18.389047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.389028    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.444905 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.444873    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:44:18.467399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.467358    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9tms\" (UniqueName: \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.467671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.467478    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.468534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.468504    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.529823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.529731    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9tms\" (UniqueName: \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\") pod \"payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.550651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.550583    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:18.569275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.569240    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.569435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.569379    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wcf\" (UniqueName: \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.670463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.670409    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.670722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.670523    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wcf\" (UniqueName: \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.671246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.671218    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.732274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.732240    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wcf\" (UniqueName: \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\") pod \"payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:18.926400 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:44:18.926358    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42733af8_2041_4ee3_b31a_31674d10ee76.slice/crio-624c4c4ad124c3599695808e64d409f60e683835701a490cbd9d512721ab2cbf WatchSource:0}: Error finding container 624c4c4ad124c3599695808e64d409f60e683835701a490cbd9d512721ab2cbf: Status 404 returned error can't find the container with id 624c4c4ad124c3599695808e64d409f60e683835701a490cbd9d512721ab2cbf
Oct 30 22:44:18.929516 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:18.929489    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:44:19.004464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.004441    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:19.381848 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:44:19.381813    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6510e6c7_e649_4743_a8dd_8ea99a26356d.slice/crio-0f8d5f1a78d8b5fe0ad7a527a08e80835089e6f1699b1f49ac1ae8c49f8aecf3 WatchSource:0}: Error finding container 0f8d5f1a78d8b5fe0ad7a527a08e80835089e6f1699b1f49ac1ae8c49f8aecf3: Status 404 returned error can't find the container with id 0f8d5f1a78d8b5fe0ad7a527a08e80835089e6f1699b1f49ac1ae8c49f8aecf3
Oct 30 22:44:19.385414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.385308    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:44:19.578661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.578620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" event={"ID":"42733af8-2041-4ee3-b31a-31674d10ee76","Type":"ContainerStarted","Data":"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"}
Oct 30 22:44:19.578840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.578691    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" event={"ID":"42733af8-2041-4ee3-b31a-31674d10ee76","Type":"ContainerStarted","Data":"624c4c4ad124c3599695808e64d409f60e683835701a490cbd9d512721ab2cbf"}
Oct 30 22:44:19.578840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.578725    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:19.580526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.580499    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" event={"ID":"6510e6c7-e649-4743-a8dd-8ea99a26356d","Type":"ContainerStarted","Data":"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"}
Oct 30 22:44:19.580655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.580559    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:19.580655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.580578    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" event={"ID":"6510e6c7-e649-4743-a8dd-8ea99a26356d","Type":"ContainerStarted","Data":"0f8d5f1a78d8b5fe0ad7a527a08e80835089e6f1699b1f49ac1ae8c49f8aecf3"}
Oct 30 22:44:19.698006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.697533    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" podStartSLOduration=1.697512997 podStartE2EDuration="1.697512997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:19.695965295 +0000 UTC m=+3982.869289719" watchObservedRunningTime="2026-10-30 22:44:19.697512997 +0000 UTC m=+3982.870837427"
Oct 30 22:44:19.814803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:19.814719    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" podStartSLOduration=1.8147009 podStartE2EDuration="1.8147009s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:19.813336539 +0000 UTC m=+3982.986660967" watchObservedRunningTime="2026-10-30 22:44:19.8147009 +0000 UTC m=+3982.988025326"
Oct 30 22:44:20.809428 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:44:20.809387    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:44:27.448369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:27.448333    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:44:30.590858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:30.590827    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:44:30.591334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:30.590897    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:44:36.384743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.384706    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:44:36.388835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.388813    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.443245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.443208    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:44:36.473595 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.473547    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.473795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.473675    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q877p\" (UniqueName: \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.489330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.489294    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-6b46c3\""
Oct 30 22:44:36.489330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.489317    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-6b46c3-dockercfg-dlhg9\""
Oct 30 22:44:36.544005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.543971    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:44:36.548605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.548569    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.575104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.575073    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q877p\" (UniqueName: \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.575297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.575220    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.576202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.576178    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.592694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.592664    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:44:36.638077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.638006    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q877p\" (UniqueName: \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\") pod \"payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.677085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.677047    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.677277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.677129    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpxx9\" (UniqueName: \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.703643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.703569    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:36.780070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.780016    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpxx9\" (UniqueName: \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.780677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.780312    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.781375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.781348    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.842541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.842501    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpxx9\" (UniqueName: \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\") pod \"payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") " pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:36.862282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:36.862247    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:37.082787 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:44:37.082756    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ab19cf1_e870_4374_afb6_5968b253956b.slice/crio-d932daf2ad287718ab8e7956c574e5f92b3d92d019aaeb87091e30deeba9756e WatchSource:0}: Error finding container d932daf2ad287718ab8e7956c574e5f92b3d92d019aaeb87091e30deeba9756e: Status 404 returned error can't find the container with id d932daf2ad287718ab8e7956c574e5f92b3d92d019aaeb87091e30deeba9756e
Oct 30 22:44:37.095015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.094971    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:44:37.259277 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:44:37.259237    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0482efe_280f_43cf_a6f7_03adcad2eb6f.slice/crio-07a589bd9e8471690dd93ff4068987d25b12bbaa9c5142812b39aa253cc1078a WatchSource:0}: Error finding container 07a589bd9e8471690dd93ff4068987d25b12bbaa9c5142812b39aa253cc1078a: Status 404 returned error can't find the container with id 07a589bd9e8471690dd93ff4068987d25b12bbaa9c5142812b39aa253cc1078a
Oct 30 22:44:37.262579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.262552    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:44:37.706848 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.706810    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" event={"ID":"3ab19cf1-e870-4374-afb6-5968b253956b","Type":"ContainerStarted","Data":"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"}
Oct 30 22:44:37.707285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.706871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" event={"ID":"3ab19cf1-e870-4374-afb6-5968b253956b","Type":"ContainerStarted","Data":"d932daf2ad287718ab8e7956c574e5f92b3d92d019aaeb87091e30deeba9756e"}
Oct 30 22:44:37.707285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.707009    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:37.708628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.708571    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" event={"ID":"a0482efe-280f-43cf-a6f7-03adcad2eb6f","Type":"ContainerStarted","Data":"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"}
Oct 30 22:44:37.708773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.708651    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:37.708773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.708674    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" event={"ID":"a0482efe-280f-43cf-a6f7-03adcad2eb6f","Type":"ContainerStarted","Data":"07a589bd9e8471690dd93ff4068987d25b12bbaa9c5142812b39aa253cc1078a"}
Oct 30 22:44:37.828227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.828175    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" podStartSLOduration=1.828156404 podStartE2EDuration="1.828156404s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:37.825024364 +0000 UTC m=+4000.998348788" watchObservedRunningTime="2026-10-30 22:44:37.828156404 +0000 UTC m=+4001.001480830"
Oct 30 22:44:37.945445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:37.945391    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" podStartSLOduration=1.9453756150000001 podStartE2EDuration="1.945375615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:44:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:44:37.941704031 +0000 UTC m=+4001.115028460" watchObservedRunningTime="2026-10-30 22:44:37.945375615 +0000 UTC m=+4001.118700042"
Oct 30 22:44:48.719571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:48.719536    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:44:48.719997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:44:48.719635    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:44:50.809463 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:44:50.809423    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:45:00.208921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.208888    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:00.215503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.215478    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:00.215669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.215625    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.220089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.220065    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:00.220212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.220186    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.224879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.224856    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.238396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.238370    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:00.243132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.243108    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.259776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.259745    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:00.265602 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.265562    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:00.265769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.265666    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.269401 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269377    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.269545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269425    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.269545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269496    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pfzr\" (UniqueName: \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.269690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269546    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jcjp\" (UniqueName: \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.269690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269570    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.269810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269722    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.269810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269743    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqm6\" (UniqueName: \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.269810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.269764    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crv2b\" (UniqueName: \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.270185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.270166    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:00.270303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.270287    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.276232 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.276204    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:00.276342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.276322    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.280743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.280720    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:00.280858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.280823    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.285031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.285006    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:45:00.285135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.285081    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:00.285218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.285202    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.289389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.289369    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:00.289517 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.289496    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.294786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.294759    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.310279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.310251    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:00.314328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.314301    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:00.319839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.319818    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:00.328620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.325286    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:00.332176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.332139    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:00.333545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.333518    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.339820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.338107    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:00.339820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.338235    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.349131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.347787    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:00.349131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.347855    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:00.354931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.352614    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:00.363363 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.363286    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:00.370636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370573    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.370816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370679    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.370816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370732    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.370980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370828    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pfzr\" (UniqueName: \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.370980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370867    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qk2n7\" (UniqueName: \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.370980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370903    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jcjp\" (UniqueName: \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.370980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370928    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.370980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370952    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.370985    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkgkj\" (UniqueName: \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371015    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371050    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9qmx\" (UniqueName: \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371098    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfzck\" (UniqueName: \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371131    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2dkf\" (UniqueName: \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371159    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.371277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371256    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2q7x\" (UniqueName: \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371295    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kxgh\" (UniqueName: \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371334    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4xlb5\" (UniqueName: \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371359    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371421    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371437    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371452    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqm6\" (UniqueName: \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371512    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crv2b\" (UniqueName: \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371569    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371619    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjt7g\" (UniqueName: \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371662    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371687    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.371782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371722    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.372477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.371897    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.372477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.372462    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.398706 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.398669    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:00.406613 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.404110    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:00.412226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.409950    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:00.415113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.415081    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:00.433239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.433205    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqm6\" (UniqueName: \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.435181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.435153    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pfzr\" (UniqueName: \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\") pod \"maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.436420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.436395    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crv2b\" (UniqueName: \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.436878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.436852    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:00.437220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.437197    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jcjp\" (UniqueName: \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\") pod \"maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.472558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472473    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.472558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472512    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjt7g\" (UniqueName: \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.472558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472534    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.472558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472551    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472570    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472643    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472690    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qk2n7\" (UniqueName: \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472719    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472735    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472757    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkgkj\" (UniqueName: \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472781    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9qmx\" (UniqueName: \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472808    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfzck\" (UniqueName: \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472828    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2dkf\" (UniqueName: \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472846    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472861    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472899    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2q7x\" (UniqueName: \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472923    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kxgh\" (UniqueName: \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.472973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.472947    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4xlb5\" (UniqueName: \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.473801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.473406    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.473801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.473445    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.473801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.473480    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.473801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.473518    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.474034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.473973    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.474175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.474154    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.474321 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.474297    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.474374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.474297    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.474374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.474306    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.531816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.531778    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:00.539488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.539460    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjt7g\" (UniqueName: \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\") pod \"maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.541424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.541377    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:00.542556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.542459    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2q7x\" (UniqueName: \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.542556 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.542478    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9qmx\" (UniqueName: \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\") pod \"maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.542782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.542715    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qk2n7\" (UniqueName: \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\") pod \"maas-api-key-cleanup-29890005-tfw6s\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.543035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.543010    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2dkf\" (UniqueName: \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\") pod \"maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.546623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.544820    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfzck\" (UniqueName: \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\") pod \"maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.546623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.544834    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kxgh\" (UniqueName: \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\") pod \"maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.546623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.545007    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4xlb5\" (UniqueName: \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\") pod \"maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.546867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.546800    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkgkj\" (UniqueName: \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\") pod \"maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.550610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.547947    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:00.557052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.557015    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:00.583828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.583798    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:00.592287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.592260    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:00.601511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.600845    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:00.610857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.610303    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:00.622473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.622265    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:00.641082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.640223    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:00.663220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.662952    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:00.676012 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.675742    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:00.684870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.684716    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:00.963009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:00.962938    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:01.176882 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.176848    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod780e1b40_53e0_4a25_9ef6_5ba366b12a6c.slice/crio-5867a1021eed16f8c1bafb54ab63fc2557b9f311191be3d8e1dc8db03e7512fe WatchSource:0}: Error finding container 5867a1021eed16f8c1bafb54ab63fc2557b9f311191be3d8e1dc8db03e7512fe: Status 404 returned error can't find the container with id 5867a1021eed16f8c1bafb54ab63fc2557b9f311191be3d8e1dc8db03e7512fe
Oct 30 22:45:01.181248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.181212    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:01.222800 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.222765    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7795d75a_dd56_486f_b664_46355f0ab91f.slice/crio-ca7828d63b7f4a1fbd038b5948ab960f4ee9005df9e736944e360c8a6d1c01b1 WatchSource:0}: Error finding container ca7828d63b7f4a1fbd038b5948ab960f4ee9005df9e736944e360c8a6d1c01b1: Status 404 returned error can't find the container with id ca7828d63b7f4a1fbd038b5948ab960f4ee9005df9e736944e360c8a6d1c01b1
Oct 30 22:45:01.224023 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.223986    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91b6445b_cb6e_4b09_bf29_e8939defc5df.slice/crio-6b2fce151ef0355efc5a6b5ed538cb391ec6b09c119ad6da0b2d35218e3a6b05 WatchSource:0}: Error finding container 6b2fce151ef0355efc5a6b5ed538cb391ec6b09c119ad6da0b2d35218e3a6b05: Status 404 returned error can't find the container with id 6b2fce151ef0355efc5a6b5ed538cb391ec6b09c119ad6da0b2d35218e3a6b05
Oct 30 22:45:01.225893 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.225812    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:01.231830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.231799    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:01.288464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.288424    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:01.292625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.292570    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:01.329113 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.329081    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f957e98_5367_4753_bed1_9cea4014f17b.slice/crio-688975b6f18a1e35b3eee7ccf284f9469f8b00bb95f5d2f8b2313ec33bf5581a WatchSource:0}: Error finding container 688975b6f18a1e35b3eee7ccf284f9469f8b00bb95f5d2f8b2313ec33bf5581a: Status 404 returned error can't find the container with id 688975b6f18a1e35b3eee7ccf284f9469f8b00bb95f5d2f8b2313ec33bf5581a
Oct 30 22:45:01.329348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.329323    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:01.331359 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.331330    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8fda827_f646_4860_b348_adb52b07ca42.slice/crio-7479302b2e028fc560a5b8f409a0f2f8c5c3db22a7340b143c4150f7158a364b WatchSource:0}: Error finding container 7479302b2e028fc560a5b8f409a0f2f8c5c3db22a7340b143c4150f7158a364b: Status 404 returned error can't find the container with id 7479302b2e028fc560a5b8f409a0f2f8c5c3db22a7340b143c4150f7158a364b
Oct 30 22:45:01.332883 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.332849    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1aeee42b_cc59_4f1d_a55b_829ba217cb0d.slice/crio-7b5d5496205a811df0ee010e5dd3128db38d2d5d6a6a9c8ea31ecc70dfc24f57 WatchSource:0}: Error finding container 7b5d5496205a811df0ee010e5dd3128db38d2d5d6a6a9c8ea31ecc70dfc24f57: Status 404 returned error can't find the container with id 7b5d5496205a811df0ee010e5dd3128db38d2d5d6a6a9c8ea31ecc70dfc24f57
Oct 30 22:45:01.334906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.334702    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:01.334906 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.334734    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20cc509b_0284_458a_8735_482f710d7f62.slice/crio-10033f325f56b7618702bac5bd781e524a6f4233b99acd0deb0e4039c7152ee8 WatchSource:0}: Error finding container 10033f325f56b7618702bac5bd781e524a6f4233b99acd0deb0e4039c7152ee8: Status 404 returned error can't find the container with id 10033f325f56b7618702bac5bd781e524a6f4233b99acd0deb0e4039c7152ee8
Oct 30 22:45:01.376755 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.376728    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a9340e4_b702_4077_8965_eb968fa24422.slice/crio-af3fb2b74e1a833b63961172470e4a6a1c07d33da9037f113e092130c59d174f WatchSource:0}: Error finding container af3fb2b74e1a833b63961172470e4a6a1c07d33da9037f113e092130c59d174f: Status 404 returned error can't find the container with id af3fb2b74e1a833b63961172470e4a6a1c07d33da9037f113e092130c59d174f
Oct 30 22:45:01.379595 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.379548    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:01.395230 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.395187    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod261c1a76_4f45_49fa_822a_b211cefef3fa.slice/crio-6a5456036faa153d2c07ce5a7d0838d5100df7f4e215c1b56a1fcc794fef65e0 WatchSource:0}: Error finding container 6a5456036faa153d2c07ce5a7d0838d5100df7f4e215c1b56a1fcc794fef65e0: Status 404 returned error can't find the container with id 6a5456036faa153d2c07ce5a7d0838d5100df7f4e215c1b56a1fcc794fef65e0
Oct 30 22:45:01.398025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.397714    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:01.432792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.432760    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:01.459696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.459666    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:01.463775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.463746    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:01.536673 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.536599    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1679375d_7196_4326_abdf_0735b3b6e814.slice/crio-34f3b877890f3b33f56275776554fab5d935442cff5aad09f5775b7d84fbc9c1 WatchSource:0}: Error finding container 34f3b877890f3b33f56275776554fab5d935442cff5aad09f5775b7d84fbc9c1: Status 404 returned error can't find the container with id 34f3b877890f3b33f56275776554fab5d935442cff5aad09f5775b7d84fbc9c1
Oct 30 22:45:01.537928 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.537543    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6379122a_c56a_4a2f_a170_597ac6eba2f1.slice/crio-65314614b3b76fadf876104bc62bebdbe499aebc6b8db47715bdec82548e2d12 WatchSource:0}: Error finding container 65314614b3b76fadf876104bc62bebdbe499aebc6b8db47715bdec82548e2d12: Status 404 returned error can't find the container with id 65314614b3b76fadf876104bc62bebdbe499aebc6b8db47715bdec82548e2d12
Oct 30 22:45:01.539179 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:45:01.538856    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1430c861_1090_42ff_a464_7bf32ad63e75.slice/crio-d3cf87dd67ee5355ada78921aedff5e03366df4aa51bd842bfe383f33e6551e9 WatchSource:0}: Error finding container d3cf87dd67ee5355ada78921aedff5e03366df4aa51bd842bfe383f33e6551e9: Status 404 returned error can't find the container with id d3cf87dd67ee5355ada78921aedff5e03366df4aa51bd842bfe383f33e6551e9
Oct 30 22:45:01.886437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.886399    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerStarted","Data":"e6620f57da6dc729da24d049ee9009d4bce2a818bcea863948ab26d98e868578"}
Oct 30 22:45:01.888650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.887037    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerStarted","Data":"7479302b2e028fc560a5b8f409a0f2f8c5c3db22a7340b143c4150f7158a364b"}
Oct 30 22:45:01.897779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.897742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerStarted","Data":"6a5456036faa153d2c07ce5a7d0838d5100df7f4e215c1b56a1fcc794fef65e0"}
Oct 30 22:45:01.899604 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.899560    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerStarted","Data":"d3cf87dd67ee5355ada78921aedff5e03366df4aa51bd842bfe383f33e6551e9"}
Oct 30 22:45:01.901263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.901240    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerStarted","Data":"f0cb72a05e83103ada07fce805fae6c907df9838d1feab5b8935208ac3a125e0"}
Oct 30 22:45:01.901383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.901278    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerStarted","Data":"688975b6f18a1e35b3eee7ccf284f9469f8b00bb95f5d2f8b2313ec33bf5581a"}
Oct 30 22:45:01.902629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.902581    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerStarted","Data":"af3fb2b74e1a833b63961172470e4a6a1c07d33da9037f113e092130c59d174f"}
Oct 30 22:45:01.903866 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.903843    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerStarted","Data":"7b5d5496205a811df0ee010e5dd3128db38d2d5d6a6a9c8ea31ecc70dfc24f57"}
Oct 30 22:45:01.905305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.905283    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerStarted","Data":"9d04e42537e5dd438979004fb0e4edb603cb83d1713d8ba11dcb6753abd0470f"}
Oct 30 22:45:01.905412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.905320    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerStarted","Data":"5867a1021eed16f8c1bafb54ab63fc2557b9f311191be3d8e1dc8db03e7512fe"}
Oct 30 22:45:01.906880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.906859    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerStarted","Data":"0a40125a4776b77a4bd81e463e2267e4f5bb8cb24c29236bbd02cabd876c9c5d"}
Oct 30 22:45:01.906978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.906895    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerStarted","Data":"fc7aabbc3abd56f9ee666b14f311d06e35fea0b9f35f3a780286d187ac835003"}
Oct 30 22:45:01.908047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.908021    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerStarted","Data":"ca7828d63b7f4a1fbd038b5948ab960f4ee9005df9e736944e360c8a6d1c01b1"}
Oct 30 22:45:01.909438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.909413    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" event={"ID":"91b6445b-cb6e-4b09-bf29-e8939defc5df","Type":"ContainerStarted","Data":"6b2fce151ef0355efc5a6b5ed538cb391ec6b09c119ad6da0b2d35218e3a6b05"}
Oct 30 22:45:01.911069 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.911021    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerStarted","Data":"34f3b877890f3b33f56275776554fab5d935442cff5aad09f5775b7d84fbc9c1"}
Oct 30 22:45:01.912522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.912498    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerStarted","Data":"65314614b3b76fadf876104bc62bebdbe499aebc6b8db47715bdec82548e2d12"}
Oct 30 22:45:01.914871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.914581    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerStarted","Data":"c3bd652dcf2cbfa6a1656db211a5d6109502aa46184ac2ebee562a2b1ad1a05e"}
Oct 30 22:45:01.915060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:01.915032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerStarted","Data":"10033f325f56b7618702bac5bd781e524a6f4233b99acd0deb0e4039c7152ee8"}
Oct 30 22:45:02.007240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.007185    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" podStartSLOduration=2.007170778 podStartE2EDuration="2.007170778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.00521549 +0000 UTC m=+4025.178539916" watchObservedRunningTime="2026-10-30 22:45:02.007170778 +0000 UTC m=+4025.180495210"
Oct 30 22:45:02.133864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.133761    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" podStartSLOduration=2.133746454 podStartE2EDuration="2.133746454s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.129869394 +0000 UTC m=+4025.303193822" watchObservedRunningTime="2026-10-30 22:45:02.133746454 +0000 UTC m=+4025.307070884"
Oct 30 22:45:02.250581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.250533    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" podStartSLOduration=2.250519576 podStartE2EDuration="2.250519576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.246365923 +0000 UTC m=+4025.419690351" watchObservedRunningTime="2026-10-30 22:45:02.250519576 +0000 UTC m=+4025.423844002"
Oct 30 22:45:02.374390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.374331    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" podStartSLOduration=2.374316127 podStartE2EDuration="2.374316127s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.373080159 +0000 UTC m=+4025.546404586" watchObservedRunningTime="2026-10-30 22:45:02.374316127 +0000 UTC m=+4025.547640608"
Oct 30 22:45:02.498634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.498556    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" podStartSLOduration=2.498541052 podStartE2EDuration="2.498541052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:02.495712315 +0000 UTC m=+4025.669036742" watchObservedRunningTime="2026-10-30 22:45:02.498541052 +0000 UTC m=+4025.671865516"
Oct 30 22:45:02.930637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.930599    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerStarted","Data":"a8d0a58cb344ae7d0c8ce2fb1a450221578803a38ab3020fb957323ae80f175e"}
Oct 30 22:45:02.932267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.932238    4882 generic.go:356] "Generic (PLEG): container finished" podID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerID="b900db007ad629215a483aa95703f92c302196544928f531e65a0aa1085858b7" exitCode=60
Oct 30 22:45:02.932418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.932320    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerDied","Data":"b900db007ad629215a483aa95703f92c302196544928f531e65a0aa1085858b7"}
Oct 30 22:45:02.932629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.932605    4882 scope.go:122] "RemoveContainer" containerID="b900db007ad629215a483aa95703f92c302196544928f531e65a0aa1085858b7"
Oct 30 22:45:02.934066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.934033    4882 generic.go:356] "Generic (PLEG): container finished" podID="20cc509b-0284-458a-8735-482f710d7f62" containerID="c3bd652dcf2cbfa6a1656db211a5d6109502aa46184ac2ebee562a2b1ad1a05e" exitCode=60
Oct 30 22:45:02.934176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.934117    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerDied","Data":"c3bd652dcf2cbfa6a1656db211a5d6109502aa46184ac2ebee562a2b1ad1a05e"}
Oct 30 22:45:02.934365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.934344    4882 scope.go:122] "RemoveContainer" containerID="c3bd652dcf2cbfa6a1656db211a5d6109502aa46184ac2ebee562a2b1ad1a05e"
Oct 30 22:45:02.936790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.936769    4882 generic.go:356] "Generic (PLEG): container finished" podID="c8fda827-f646-4860-b348-adb52b07ca42" containerID="e6620f57da6dc729da24d049ee9009d4bce2a818bcea863948ab26d98e868578" exitCode=60
Oct 30 22:45:02.937081 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.937066    4882 scope.go:122] "RemoveContainer" containerID="e6620f57da6dc729da24d049ee9009d4bce2a818bcea863948ab26d98e868578"
Oct 30 22:45:02.938981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.938953    4882 generic.go:356] "Generic (PLEG): container finished" podID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerID="4b1d141ae282c9194f772d148b723391824b3fb589b7a24efd1c7e864531e597" exitCode=60
Oct 30 22:45:02.939402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.939384    4882 scope.go:122] "RemoveContainer" containerID="4b1d141ae282c9194f772d148b723391824b3fb589b7a24efd1c7e864531e597"
Oct 30 22:45:02.941479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.941441    4882 generic.go:356] "Generic (PLEG): container finished" podID="1430c861-1090-42ff-a464-7bf32ad63e75" containerID="18c86337fc99ff3ddfdb6ce7650039dcdfebc47855b3fbd5117364d80632fbe1" exitCode=60
Oct 30 22:45:02.942761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.942730    4882 scope.go:122] "RemoveContainer" containerID="18c86337fc99ff3ddfdb6ce7650039dcdfebc47855b3fbd5117364d80632fbe1"
Oct 30 22:45:02.945151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.944989    4882 generic.go:356] "Generic (PLEG): container finished" podID="5f957e98-5367-4753-bed1-9cea4014f17b" containerID="f0cb72a05e83103ada07fce805fae6c907df9838d1feab5b8935208ac3a125e0" exitCode=60
Oct 30 22:45:02.945367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.945349    4882 scope.go:122] "RemoveContainer" containerID="f0cb72a05e83103ada07fce805fae6c907df9838d1feab5b8935208ac3a125e0"
Oct 30 22:45:02.948092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.948072    4882 generic.go:356] "Generic (PLEG): container finished" podID="2a9340e4-b702-4077-8965-eb968fa24422" containerID="50b76ad96b5df64bfef9998dd42ba92578982bef4a24f94972cd90a83a5fec09" exitCode=60
Oct 30 22:45:02.948412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.948391    4882 scope.go:122] "RemoveContainer" containerID="50b76ad96b5df64bfef9998dd42ba92578982bef4a24f94972cd90a83a5fec09"
Oct 30 22:45:02.949653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.949577    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerDied","Data":"e6620f57da6dc729da24d049ee9009d4bce2a818bcea863948ab26d98e868578"}
Oct 30 22:45:02.949741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.949690    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerDied","Data":"4b1d141ae282c9194f772d148b723391824b3fb589b7a24efd1c7e864531e597"}
Oct 30 22:45:02.949741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.949722    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerDied","Data":"18c86337fc99ff3ddfdb6ce7650039dcdfebc47855b3fbd5117364d80632fbe1"}
Oct 30 22:45:02.949882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.949748    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerDied","Data":"f0cb72a05e83103ada07fce805fae6c907df9838d1feab5b8935208ac3a125e0"}
Oct 30 22:45:02.949882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.949774    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerDied","Data":"50b76ad96b5df64bfef9998dd42ba92578982bef4a24f94972cd90a83a5fec09"}
Oct 30 22:45:02.950203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.950176    4882 generic.go:356] "Generic (PLEG): container finished" podID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerID="b68dfbee6c0542a0bd3905bbfd1891c90caaffe71e9804549e59cfea85b0dff9" exitCode=60
Oct 30 22:45:02.950416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.950394    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerDied","Data":"b68dfbee6c0542a0bd3905bbfd1891c90caaffe71e9804549e59cfea85b0dff9"}
Oct 30 22:45:02.950526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.950508    4882 scope.go:122] "RemoveContainer" containerID="b68dfbee6c0542a0bd3905bbfd1891c90caaffe71e9804549e59cfea85b0dff9"
Oct 30 22:45:02.952252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.952229    4882 generic.go:356] "Generic (PLEG): container finished" podID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerID="9d04e42537e5dd438979004fb0e4edb603cb83d1713d8ba11dcb6753abd0470f" exitCode=60
Oct 30 22:45:02.952351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.952312    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerDied","Data":"9d04e42537e5dd438979004fb0e4edb603cb83d1713d8ba11dcb6753abd0470f"}
Oct 30 22:45:02.952525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.952500    4882 scope.go:122] "RemoveContainer" containerID="9d04e42537e5dd438979004fb0e4edb603cb83d1713d8ba11dcb6753abd0470f"
Oct 30 22:45:02.959181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.959024    4882 generic.go:356] "Generic (PLEG): container finished" podID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerID="0a40125a4776b77a4bd81e463e2267e4f5bb8cb24c29236bbd02cabd876c9c5d" exitCode=60
Oct 30 22:45:02.959181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.959128    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerDied","Data":"0a40125a4776b77a4bd81e463e2267e4f5bb8cb24c29236bbd02cabd876c9c5d"}
Oct 30 22:45:02.959680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.959461    4882 scope.go:122] "RemoveContainer" containerID="0a40125a4776b77a4bd81e463e2267e4f5bb8cb24c29236bbd02cabd876c9c5d"
Oct 30 22:45:02.961477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.961451    4882 generic.go:356] "Generic (PLEG): container finished" podID="7795d75a-dd56-486f-b664-46355f0ab91f" containerID="7408d27b370e39a0bb7ddc9e85e8043cee43319b8f7662a6b999f3955adfa18f" exitCode=60
Oct 30 22:45:02.961640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.961532    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerDied","Data":"7408d27b370e39a0bb7ddc9e85e8043cee43319b8f7662a6b999f3955adfa18f"}
Oct 30 22:45:02.961921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.961733    4882 scope.go:122] "RemoveContainer" containerID="7408d27b370e39a0bb7ddc9e85e8043cee43319b8f7662a6b999f3955adfa18f"
Oct 30 22:45:02.963482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.963458    4882 generic.go:356] "Generic (PLEG): container finished" podID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerID="b9cb2ba396273fbeb16f3048d21e35afb00fcdc0f6de912889ec5c41b177a10e" exitCode=60
Oct 30 22:45:02.963575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.963490    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" event={"ID":"91b6445b-cb6e-4b09-bf29-e8939defc5df","Type":"ContainerDied","Data":"b9cb2ba396273fbeb16f3048d21e35afb00fcdc0f6de912889ec5c41b177a10e"}
Oct 30 22:45:02.963726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:02.963711    4882 scope.go:122] "RemoveContainer" containerID="b9cb2ba396273fbeb16f3048d21e35afb00fcdc0f6de912889ec5c41b177a10e"
Oct 30 22:45:03.051099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.051049    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" podStartSLOduration=3.051034356 podStartE2EDuration="3.051034356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:03.04682475 +0000 UTC m=+4026.220149176" watchObservedRunningTime="2026-10-30 22:45:03.051034356 +0000 UTC m=+4026.224358782"
Oct 30 22:45:03.973552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.973519    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerStarted","Data":"501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"}
Oct 30 22:45:03.975774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.975731    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerStarted","Data":"88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"}
Oct 30 22:45:03.977771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.977743    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerStarted","Data":"e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"}
Oct 30 22:45:03.979632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.979572    4882 generic.go:356] "Generic (PLEG): container finished" podID="1679375d-7196-4326-abdf-0735b3b6e814" containerID="a8d0a58cb344ae7d0c8ce2fb1a450221578803a38ab3020fb957323ae80f175e" exitCode=60
Oct 30 22:45:03.979632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.979618    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerDied","Data":"a8d0a58cb344ae7d0c8ce2fb1a450221578803a38ab3020fb957323ae80f175e"}
Oct 30 22:45:03.979933 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.979916    4882 scope.go:122] "RemoveContainer" containerID="a8d0a58cb344ae7d0c8ce2fb1a450221578803a38ab3020fb957323ae80f175e"
Oct 30 22:45:03.981954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.981933    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerStarted","Data":"7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"}
Oct 30 22:45:03.984039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.984014    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerStarted","Data":"adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"}
Oct 30 22:45:03.991319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.991290    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerStarted","Data":"7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"}
Oct 30 22:45:03.993412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.993384    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerStarted","Data":"4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"}
Oct 30 22:45:03.995622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.995554    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerStarted","Data":"f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"}
Oct 30 22:45:03.997472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.997446    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerStarted","Data":"194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"}
Oct 30 22:45:03.999422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:03.999384    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerStarted","Data":"6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"}
Oct 30 22:45:04.001289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:04.001266    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerStarted","Data":"d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"}
Oct 30 22:45:04.572329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:04.572249    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" podStartSLOduration=4.572227947 podStartE2EDuration="4.572227947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.567439093 +0000 UTC m=+4027.740763519" watchObservedRunningTime="2026-10-30 22:45:04.572227947 +0000 UTC m=+4027.745552377"
Oct 30 22:45:04.683508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:04.683448    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" podStartSLOduration=4.6834303120000005 podStartE2EDuration="4.683430312s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.683019815 +0000 UTC m=+4027.856344242" watchObservedRunningTime="2026-10-30 22:45:04.683430312 +0000 UTC m=+4027.856754742"
Oct 30 22:45:04.802341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:04.802275    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" podStartSLOduration=4.802260949 podStartE2EDuration="4.802260949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.798896519 +0000 UTC m=+4027.972220947" watchObservedRunningTime="2026-10-30 22:45:04.802260949 +0000 UTC m=+4027.975585375"
Oct 30 22:45:04.924395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:04.924342    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" podStartSLOduration=4.924326935 podStartE2EDuration="4.924326935s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:04.922357962 +0000 UTC m=+4028.095682389" watchObservedRunningTime="2026-10-30 22:45:04.924326935 +0000 UTC m=+4028.097651362"
Oct 30 22:45:05.010173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.010120    4882 generic.go:356] "Generic (PLEG): container finished" podID="5f957e98-5367-4753-bed1-9cea4014f17b" containerID="194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83" exitCode=60
Oct 30 22:45:05.010642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.010175    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerDied","Data":"194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"}
Oct 30 22:45:05.010642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.010247    4882 scope.go:122] "RemoveContainer" containerID="f0cb72a05e83103ada07fce805fae6c907df9838d1feab5b8935208ac3a125e0"
Oct 30 22:45:05.010642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.010522    4882 scope.go:122] "RemoveContainer" containerID="194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"
Oct 30 22:45:05.010920 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.010893    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-tfw6s_odh-ai-gateway-infra(5f957e98-5367-4753-bed1-9cea4014f17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" podUID="5f957e98-5367-4753-bed1-9cea4014f17b"
Oct 30 22:45:05.012540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.012519    4882 generic.go:356] "Generic (PLEG): container finished" podID="2a9340e4-b702-4077-8965-eb968fa24422" containerID="6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f" exitCode=60
Oct 30 22:45:05.012687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.012622    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerDied","Data":"6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"}
Oct 30 22:45:05.012884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.012869    4882 scope.go:122] "RemoveContainer" containerID="6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"
Oct 30 22:45:05.013202 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.013182    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5_odh-ai-gateway-infra(2a9340e4-b702-4077-8965-eb968fa24422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" podUID="2a9340e4-b702-4077-8965-eb968fa24422"
Oct 30 22:45:05.014731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.014709    4882 generic.go:356] "Generic (PLEG): container finished" podID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerID="d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b" exitCode=60
Oct 30 22:45:05.014833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.014773    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerDied","Data":"d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"}
Oct 30 22:45:05.015071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.015055    4882 scope.go:122] "RemoveContainer" containerID="d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"
Oct 30 22:45:05.015365 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.015345    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn_odh-ai-gateway-infra(1aeee42b-cc59-4f1d-a55b-829ba217cb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d"
Oct 30 22:45:05.017026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.017005    4882 generic.go:356] "Generic (PLEG): container finished" podID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerID="501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2" exitCode=60
Oct 30 22:45:05.017116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.017084    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerDied","Data":"501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"}
Oct 30 22:45:05.017293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.017278    4882 scope.go:122] "RemoveContainer" containerID="501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"
Oct 30 22:45:05.017550 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.017531    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt_odh-ai-gateway-infra(780e1b40-53e0-4a25-9ef6-5ba366b12a6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c"
Oct 30 22:45:05.019133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.019110    4882 generic.go:356] "Generic (PLEG): container finished" podID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerID="88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2" exitCode=60
Oct 30 22:45:05.019240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.019185    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerDied","Data":"88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"}
Oct 30 22:45:05.019404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.019386    4882 scope.go:122] "RemoveContainer" containerID="88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"
Oct 30 22:45:05.019692 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.019667    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg_odh-ai-gateway-infra(c59e7d31-e436-4408-b2dc-755f0b57d1ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed"
Oct 30 22:45:05.021356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.021334    4882 generic.go:356] "Generic (PLEG): container finished" podID="7795d75a-dd56-486f-b664-46355f0ab91f" containerID="e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc" exitCode=60
Oct 30 22:45:05.021460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.021373    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerDied","Data":"e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"}
Oct 30 22:45:05.021679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.021658    4882 scope.go:122] "RemoveContainer" containerID="e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"
Oct 30 22:45:05.021980 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.021959    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv_odh-ai-gateway-infra(7795d75a-dd56-486f-b664-46355f0ab91f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" podUID="7795d75a-dd56-486f-b664-46355f0ab91f"
Oct 30 22:45:05.023785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.023764    4882 generic.go:356] "Generic (PLEG): container finished" podID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerID="27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4" exitCode=60
Oct 30 22:45:05.023885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.023843    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" event={"ID":"91b6445b-cb6e-4b09-bf29-e8939defc5df","Type":"ContainerDied","Data":"27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4"}
Oct 30 22:45:05.024086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.024071    4882 scope.go:122] "RemoveContainer" containerID="27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4"
Oct 30 22:45:05.024349 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.024323    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8_odh-ai-gateway-infra(91b6445b-cb6e-4b09-bf29-e8939defc5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df"
Oct 30 22:45:05.025854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.025832    4882 generic.go:356] "Generic (PLEG): container finished" podID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerID="7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5" exitCode=60
Oct 30 22:45:05.025980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.025854    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerDied","Data":"7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"}
Oct 30 22:45:05.026149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.026130    4882 scope.go:122] "RemoveContainer" containerID="7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"
Oct 30 22:45:05.026417 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.026393    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww_odh-ai-gateway-infra(6379122a-c56a-4a2f-a170-597ac6eba2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1"
Oct 30 22:45:05.027060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.027032    4882 scope.go:122] "RemoveContainer" containerID="50b76ad96b5df64bfef9998dd42ba92578982bef4a24f94972cd90a83a5fec09"
Oct 30 22:45:05.028384 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.028364    4882 generic.go:356] "Generic (PLEG): container finished" podID="20cc509b-0284-458a-8735-482f710d7f62" containerID="adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c" exitCode=60
Oct 30 22:45:05.028482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.028454    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerDied","Data":"adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"}
Oct 30 22:45:05.028720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.028704    4882 scope.go:122] "RemoveContainer" containerID="adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"
Oct 30 22:45:05.029162 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.028960    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8_odh-ai-gateway-infra(20cc509b-0284-458a-8735-482f710d7f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" podUID="20cc509b-0284-458a-8735-482f710d7f62"
Oct 30 22:45:05.030909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.030885    4882 generic.go:356] "Generic (PLEG): container finished" podID="c8fda827-f646-4860-b348-adb52b07ca42" containerID="7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda" exitCode=60
Oct 30 22:45:05.030989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.030967    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerDied","Data":"7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"}
Oct 30 22:45:05.031204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.031185    4882 scope.go:122] "RemoveContainer" containerID="7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"
Oct 30 22:45:05.031452 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.031430    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk_odh-ai-gateway-infra(c8fda827-f646-4860-b348-adb52b07ca42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" podUID="c8fda827-f646-4860-b348-adb52b07ca42"
Oct 30 22:45:05.034873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.034767    4882 generic.go:356] "Generic (PLEG): container finished" podID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerID="4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad" exitCode=60
Oct 30 22:45:05.034873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.034853    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerDied","Data":"4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"}
Oct 30 22:45:05.035079 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.035063    4882 scope.go:122] "RemoveContainer" containerID="4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"
Oct 30 22:45:05.035313 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.035286    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr_odh-ai-gateway-infra(261c1a76-4f45-49fa-822a-b211cefef3fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa"
Oct 30 22:45:05.037389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.037367    4882 generic.go:356] "Generic (PLEG): container finished" podID="1430c861-1090-42ff-a464-7bf32ad63e75" containerID="f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829" exitCode=60
Oct 30 22:45:05.037508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.037412    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerDied","Data":"f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"}
Oct 30 22:45:05.037651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.037635    4882 scope.go:122] "RemoveContainer" containerID="f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"
Oct 30 22:45:05.037907 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:05.037882    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc_odh-ai-gateway-infra(1430c861-1090-42ff-a464-7bf32ad63e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" podUID="1430c861-1090-42ff-a464-7bf32ad63e75"
Oct 30 22:45:05.043457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.043354    4882 scope.go:122] "RemoveContainer" containerID="b68dfbee6c0542a0bd3905bbfd1891c90caaffe71e9804549e59cfea85b0dff9"
Oct 30 22:45:05.056741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.056717    4882 scope.go:122] "RemoveContainer" containerID="9d04e42537e5dd438979004fb0e4edb603cb83d1713d8ba11dcb6753abd0470f"
Oct 30 22:45:05.070842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.070820    4882 scope.go:122] "RemoveContainer" containerID="0a40125a4776b77a4bd81e463e2267e4f5bb8cb24c29236bbd02cabd876c9c5d"
Oct 30 22:45:05.084699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.084673    4882 scope.go:122] "RemoveContainer" containerID="7408d27b370e39a0bb7ddc9e85e8043cee43319b8f7662a6b999f3955adfa18f"
Oct 30 22:45:05.098815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.098792    4882 scope.go:122] "RemoveContainer" containerID="b9cb2ba396273fbeb16f3048d21e35afb00fcdc0f6de912889ec5c41b177a10e"
Oct 30 22:45:05.112637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.112617    4882 scope.go:122] "RemoveContainer" containerID="b900db007ad629215a483aa95703f92c302196544928f531e65a0aa1085858b7"
Oct 30 22:45:05.126283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.126259    4882 scope.go:122] "RemoveContainer" containerID="c3bd652dcf2cbfa6a1656db211a5d6109502aa46184ac2ebee562a2b1ad1a05e"
Oct 30 22:45:05.145567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.144840    4882 scope.go:122] "RemoveContainer" containerID="e6620f57da6dc729da24d049ee9009d4bce2a818bcea863948ab26d98e868578"
Oct 30 22:45:05.161383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.161358    4882 scope.go:122] "RemoveContainer" containerID="4b1d141ae282c9194f772d148b723391824b3fb589b7a24efd1c7e864531e597"
Oct 30 22:45:05.175379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.175351    4882 scope.go:122] "RemoveContainer" containerID="18c86337fc99ff3ddfdb6ce7650039dcdfebc47855b3fbd5117364d80632fbe1"
Oct 30 22:45:05.276262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.276204    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" podStartSLOduration=5.276188517 podStartE2EDuration="5.276188517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.273997184 +0000 UTC m=+4028.447321612" watchObservedRunningTime="2026-10-30 22:45:05.276188517 +0000 UTC m=+4028.449512944"
Oct 30 22:45:05.895188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:05.895132    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" podStartSLOduration=5.895115731 podStartE2EDuration="5.895115731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:05.892631194 +0000 UTC m=+4029.065955618" watchObservedRunningTime="2026-10-30 22:45:05.895115731 +0000 UTC m=+4029.068440208"
Oct 30 22:45:06.047447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.047420    4882 scope.go:122] "RemoveContainer" containerID="88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"
Oct 30 22:45:06.047917 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.047776    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg_odh-ai-gateway-infra(c59e7d31-e436-4408-b2dc-755f0b57d1ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed"
Oct 30 22:45:06.049155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.049126    4882 scope.go:122] "RemoveContainer" containerID="e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"
Oct 30 22:45:06.049468 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.049438    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv_odh-ai-gateway-infra(7795d75a-dd56-486f-b664-46355f0ab91f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" podUID="7795d75a-dd56-486f-b664-46355f0ab91f"
Oct 30 22:45:06.050777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.050760    4882 scope.go:122] "RemoveContainer" containerID="27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4"
Oct 30 22:45:06.051066 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.051030    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8_odh-ai-gateway-infra(91b6445b-cb6e-4b09-bf29-e8939defc5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df"
Oct 30 22:45:06.052394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.052370    4882 generic.go:356] "Generic (PLEG): container finished" podID="1679375d-7196-4326-abdf-0735b3b6e814" containerID="2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc" exitCode=60
Oct 30 22:45:06.052497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.052452    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerDied","Data":"2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc"}
Oct 30 22:45:06.052571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.052529    4882 scope.go:122] "RemoveContainer" containerID="a8d0a58cb344ae7d0c8ce2fb1a450221578803a38ab3020fb957323ae80f175e"
Oct 30 22:45:06.052741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.052724    4882 scope.go:122] "RemoveContainer" containerID="2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc"
Oct 30 22:45:06.053044 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.053021    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp_odh-ai-gateway-infra(1679375d-7196-4326-abdf-0735b3b6e814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" podUID="1679375d-7196-4326-abdf-0735b3b6e814"
Oct 30 22:45:06.054435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.054418    4882 scope.go:122] "RemoveContainer" containerID="7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"
Oct 30 22:45:06.054730 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.054705    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww_odh-ai-gateway-infra(6379122a-c56a-4a2f-a170-597ac6eba2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1"
Oct 30 22:45:06.056147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.056129    4882 scope.go:122] "RemoveContainer" containerID="adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"
Oct 30 22:45:06.056362 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.056340    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8_odh-ai-gateway-infra(20cc509b-0284-458a-8735-482f710d7f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" podUID="20cc509b-0284-458a-8735-482f710d7f62"
Oct 30 22:45:06.057986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.057969    4882 scope.go:122] "RemoveContainer" containerID="7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"
Oct 30 22:45:06.058204 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.058182    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk_odh-ai-gateway-infra(c8fda827-f646-4860-b348-adb52b07ca42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" podUID="c8fda827-f646-4860-b348-adb52b07ca42"
Oct 30 22:45:06.059579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.059562    4882 scope.go:122] "RemoveContainer" containerID="4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"
Oct 30 22:45:06.059796 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.059775    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr_odh-ai-gateway-infra(261c1a76-4f45-49fa-822a-b211cefef3fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa"
Oct 30 22:45:06.061296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.061279    4882 scope.go:122] "RemoveContainer" containerID="f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"
Oct 30 22:45:06.061494 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.061476    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc_odh-ai-gateway-infra(1430c861-1090-42ff-a464-7bf32ad63e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" podUID="1430c861-1090-42ff-a464-7bf32ad63e75"
Oct 30 22:45:06.063005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.062986    4882 scope.go:122] "RemoveContainer" containerID="194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"
Oct 30 22:45:06.063222 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.063178    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-tfw6s_odh-ai-gateway-infra(5f957e98-5367-4753-bed1-9cea4014f17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" podUID="5f957e98-5367-4753-bed1-9cea4014f17b"
Oct 30 22:45:06.064625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.064577    4882 scope.go:122] "RemoveContainer" containerID="6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"
Oct 30 22:45:06.064836 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.064813    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5_odh-ai-gateway-infra(2a9340e4-b702-4077-8965-eb968fa24422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" podUID="2a9340e4-b702-4077-8965-eb968fa24422"
Oct 30 22:45:06.066361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.066346    4882 scope.go:122] "RemoveContainer" containerID="d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"
Oct 30 22:45:06.066521 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.066505    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn_odh-ai-gateway-infra(1aeee42b-cc59-4f1d-a55b-829ba217cb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d"
Oct 30 22:45:06.068005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:06.067991    4882 scope.go:122] "RemoveContainer" containerID="501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"
Oct 30 22:45:06.068176 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:06.068160    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt_odh-ai-gateway-infra(780e1b40-53e0-4a25-9ef6-5ba366b12a6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c"
Oct 30 22:45:07.077053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:07.077009    4882 scope.go:122] "RemoveContainer" containerID="2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc"
Oct 30 22:45:07.077457 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:07.077331    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp_odh-ai-gateway-infra(1679375d-7196-4326-abdf-0735b3b6e814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" podUID="1679375d-7196-4326-abdf-0735b3b6e814"
Oct 30 22:45:09.053434 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:09.053397    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"]
Oct 30 22:45:09.057628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:09.057596    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c9553a0b-78kh9"]
Oct 30 22:45:09.819958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:09.810333    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:45:10.944615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:10.944558    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4601e620-e922-444f-90b6-4c435ec9a1ca" path="/var/lib/kubelet/pods/4601e620-e922-444f-90b6-4c435ec9a1ca/volumes"
Oct 30 22:45:14.936194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:14.936153    4882 scope.go:122] "RemoveContainer" containerID="d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"
Oct 30 22:45:15.936362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:15.936332    4882 scope.go:122] "RemoveContainer" containerID="4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"
Oct 30 22:45:15.936822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:15.936521    4882 scope.go:122] "RemoveContainer" containerID="7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"
Oct 30 22:45:16.143874 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.143842    4882 generic.go:356] "Generic (PLEG): container finished" podID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerID="3bff3d1abf3e1ac24893f8ebf82d6af7418029139408e8a8c2b2b8e65d792262" exitCode=60
Oct 30 22:45:16.144094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.143916    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerDied","Data":"3bff3d1abf3e1ac24893f8ebf82d6af7418029139408e8a8c2b2b8e65d792262"}
Oct 30 22:45:16.144094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.143994    4882 scope.go:122] "RemoveContainer" containerID="d090dd9c1104b2205d1fc0ba3670e0ef63c30bcd6fc1ec361095ee32ff6cdd2b"
Oct 30 22:45:16.144451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.144433    4882 scope.go:122] "RemoveContainer" containerID="3bff3d1abf3e1ac24893f8ebf82d6af7418029139408e8a8c2b2b8e65d792262"
Oct 30 22:45:16.144807 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:16.144774    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn_odh-ai-gateway-infra(1aeee42b-cc59-4f1d-a55b-829ba217cb0d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d"
Oct 30 22:45:16.942554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.942521    4882 scope.go:122] "RemoveContainer" containerID="27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4"
Oct 30 22:45:16.942974 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:16.942654    4882 scope.go:122] "RemoveContainer" containerID="194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"
Oct 30 22:45:17.153951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.153918    4882 generic.go:356] "Generic (PLEG): container finished" podID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerID="848d27b3cf989038e571c32cf4efd77431894da8d8ccc72d8e33935ef7286a78" exitCode=60
Oct 30 22:45:17.154151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.154001    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerDied","Data":"848d27b3cf989038e571c32cf4efd77431894da8d8ccc72d8e33935ef7286a78"}
Oct 30 22:45:17.154151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.154063    4882 scope.go:122] "RemoveContainer" containerID="7aaadab3b10cf665b3e81c1a0fd3a2c18e550b15bdde7ef97ba76ef7153921a5"
Oct 30 22:45:17.154509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.154488    4882 scope.go:122] "RemoveContainer" containerID="848d27b3cf989038e571c32cf4efd77431894da8d8ccc72d8e33935ef7286a78"
Oct 30 22:45:17.155227 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:17.154878    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww_odh-ai-gateway-infra(6379122a-c56a-4a2f-a170-597ac6eba2f1)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1"
Oct 30 22:45:17.156215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.156194    4882 generic.go:356] "Generic (PLEG): container finished" podID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerID="6edd0f963c4b12ef1529fc0aff87dd805b9d2f805c46afc9a3679a292f6e3e20" exitCode=60
Oct 30 22:45:17.156311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.156261    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerDied","Data":"6edd0f963c4b12ef1529fc0aff87dd805b9d2f805c46afc9a3679a292f6e3e20"}
Oct 30 22:45:17.156669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.156650    4882 scope.go:122] "RemoveContainer" containerID="6edd0f963c4b12ef1529fc0aff87dd805b9d2f805c46afc9a3679a292f6e3e20"
Oct 30 22:45:17.156995 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:17.156969    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr_odh-ai-gateway-infra(261c1a76-4f45-49fa-822a-b211cefef3fa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa"
Oct 30 22:45:17.168929 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.168908    4882 scope.go:122] "RemoveContainer" containerID="4f877499c9f38727c2555e5d7254c972a6840173b08b74d68e36d6565c62c1ad"
Oct 30 22:45:17.273616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.273570    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:17.446666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.446637    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:17.597606 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.597563    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\" (UniqueName: \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\") pod \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") "
Oct 30 22:45:17.597808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.597659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\") pod \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\" (UID: \"1aeee42b-cc59-4f1d-a55b-829ba217cb0d\") "
Oct 30 22:45:17.598034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.598004    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca" pod "1aeee42b-cc59-4f1d-a55b-829ba217cb0d" (UID: "1aeee42b-cc59-4f1d-a55b-829ba217cb0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:17.598256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.598232    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.601492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.601140    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp" pod "1aeee42b-cc59-4f1d-a55b-829ba217cb0d" (UID: "1aeee42b-cc59-4f1d-a55b-829ba217cb0d"). InnerVolumeSpecName "kube-api-access-6jcjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:17.699687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.699567    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6jcjp\" (UniqueName: \"kubernetes.io/projected/1aeee42b-cc59-4f1d-a55b-829ba217cb0d-kube-api-access-6jcjp\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:17.936174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.936146    4882 scope.go:122] "RemoveContainer" containerID="e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"
Oct 30 22:45:17.936337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:17.936237    4882 scope.go:122] "RemoveContainer" containerID="501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"
Oct 30 22:45:18.167554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.167519    4882 generic.go:356] "Generic (PLEG): container finished" podID="5f957e98-5367-4753-bed1-9cea4014f17b" containerID="1c40c6967f4fd41d739a71ae85cd96d2e2ecb7523fc55d4787ed866447ba3c39" exitCode=60
Oct 30 22:45:18.168037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.167621    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerDied","Data":"1c40c6967f4fd41d739a71ae85cd96d2e2ecb7523fc55d4787ed866447ba3c39"}
Oct 30 22:45:18.168037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.167697    4882 scope.go:122] "RemoveContainer" containerID="194a83002615214dc8888d7fb04442857c637ba94c83c7b17ae15a8683134d83"
Oct 30 22:45:18.168139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.168120    4882 scope.go:122] "RemoveContainer" containerID="1c40c6967f4fd41d739a71ae85cd96d2e2ecb7523fc55d4787ed866447ba3c39"
Oct 30 22:45:18.168511 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:18.168482    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29890005-tfw6s_odh-ai-gateway-infra(5f957e98-5367-4753-bed1-9cea4014f17b)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" podUID="5f957e98-5367-4753-bed1-9cea4014f17b"
Oct 30 22:45:18.169185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.169169    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"
Oct 30 22:45:18.169288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.169216    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn" event={"ID":"1aeee42b-cc59-4f1d-a55b-829ba217cb0d","Type":"ContainerDied","Data":"7b5d5496205a811df0ee010e5dd3128db38d2d5d6a6a9c8ea31ecc70dfc24f57"}
Oct 30 22:45:18.171238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.171217    4882 generic.go:356] "Generic (PLEG): container finished" podID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerID="796662be1205ad6897fd28181bd7fc9ad9864f53fdd7b82aa907c10c46bdd146" exitCode=60
Oct 30 22:45:18.171353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.171259    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" event={"ID":"91b6445b-cb6e-4b09-bf29-e8939defc5df","Type":"ContainerDied","Data":"796662be1205ad6897fd28181bd7fc9ad9864f53fdd7b82aa907c10c46bdd146"}
Oct 30 22:45:18.171694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.171677    4882 scope.go:122] "RemoveContainer" containerID="796662be1205ad6897fd28181bd7fc9ad9864f53fdd7b82aa907c10c46bdd146"
Oct 30 22:45:18.172025 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:18.171999    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8_odh-ai-gateway-infra(91b6445b-cb6e-4b09-bf29-e8939defc5df)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df"
Oct 30 22:45:18.188399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.188377    4882 scope.go:122] "RemoveContainer" containerID="3bff3d1abf3e1ac24893f8ebf82d6af7418029139408e8a8c2b2b8e65d792262"
Oct 30 22:45:18.200599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.200562    4882 scope.go:122] "RemoveContainer" containerID="27cb9802f4878bca6f41ae78c2d8ed5ab2e86255c22d49a5fb65c260440a76e4"
Oct 30 22:45:18.287165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.287119    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:18.406885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.406849    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:18.415144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.415105    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:18.420968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.420944    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w1-ef00a4-29890005-kg5vn"]
Oct 30 22:45:18.468019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.467863    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:18.517703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.517670    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\") pod \"6379122a-c56a-4a2f-a170-597ac6eba2f1\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") "
Oct 30 22:45:18.517887 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.517715    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\" (UniqueName: \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\") pod \"6379122a-c56a-4a2f-a170-597ac6eba2f1\" (UID: \"6379122a-c56a-4a2f-a170-597ac6eba2f1\") "
Oct 30 22:45:18.518078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.518047    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca" pod "6379122a-c56a-4a2f-a170-597ac6eba2f1" (UID: "6379122a-c56a-4a2f-a170-597ac6eba2f1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.518605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.518363    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6379122a-c56a-4a2f-a170-597ac6eba2f1-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.520068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.520036    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh" pod "6379122a-c56a-4a2f-a170-597ac6eba2f1" (UID: "6379122a-c56a-4a2f-a170-597ac6eba2f1"). InnerVolumeSpecName "kube-api-access-8kxgh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.560637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.560576    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:18.619184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.619148    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\" (UniqueName: \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\") pod \"261c1a76-4f45-49fa-822a-b211cefef3fa\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") "
Oct 30 22:45:18.619368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.619295    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\") pod \"261c1a76-4f45-49fa-822a-b211cefef3fa\" (UID: \"261c1a76-4f45-49fa-822a-b211cefef3fa\") "
Oct 30 22:45:18.619685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.619659    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca" pod "261c1a76-4f45-49fa-822a-b211cefef3fa" (UID: "261c1a76-4f45-49fa-822a-b211cefef3fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:18.619769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.619720    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kxgh\" (UniqueName: \"kubernetes.io/projected/6379122a-c56a-4a2f-a170-597ac6eba2f1-kube-api-access-8kxgh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.621402 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.621378    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g" pod "261c1a76-4f45-49fa-822a-b211cefef3fa" (UID: "261c1a76-4f45-49fa-822a-b211cefef3fa"). InnerVolumeSpecName "kube-api-access-zjt7g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:18.721189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.721105    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjt7g\" (UniqueName: \"kubernetes.io/projected/261c1a76-4f45-49fa-822a-b211cefef3fa-kube-api-access-zjt7g\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.721189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.721136    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/261c1a76-4f45-49fa-822a-b211cefef3fa-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:18.936351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.936308    4882 scope.go:122] "RemoveContainer" containerID="f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"
Oct 30 22:45:18.936525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.936504    4882 scope.go:122] "RemoveContainer" containerID="88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"
Oct 30 22:45:18.936818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.936796    4882 scope.go:122] "RemoveContainer" containerID="7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"
Oct 30 22:45:18.937410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.937105    4882 scope.go:122] "RemoveContainer" containerID="adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"
Oct 30 22:45:18.938684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.938644    4882 scope.go:122] "RemoveContainer" containerID="6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"
Oct 30 22:45:18.948894 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:18.948861    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" path="/var/lib/kubelet/pods/1aeee42b-cc59-4f1d-a55b-829ba217cb0d/volumes"
Oct 30 22:45:19.055089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.055062    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"]
Oct 30 22:45:19.058915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.058891    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b25a8634-jjxms"]
Oct 30 22:45:19.184701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.184666    4882 generic.go:356] "Generic (PLEG): container finished" podID="7795d75a-dd56-486f-b664-46355f0ab91f" containerID="8342b7fcadeecab4fbd34b9900088d61cca019452cf3a057d566c1d4f22abaa3" exitCode=60
Oct 30 22:45:19.185105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.184741    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerDied","Data":"8342b7fcadeecab4fbd34b9900088d61cca019452cf3a057d566c1d4f22abaa3"}
Oct 30 22:45:19.185105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.184817    4882 scope.go:122] "RemoveContainer" containerID="e88a3a717b1da3efeec737d373550558cd6a4445425ffa35c6de95ec088875bc"
Oct 30 22:45:19.185287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.185221    4882 scope.go:122] "RemoveContainer" containerID="8342b7fcadeecab4fbd34b9900088d61cca019452cf3a057d566c1d4f22abaa3"
Oct 30 22:45:19.185611 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:19.185569    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv_odh-ai-gateway-infra(7795d75a-dd56-486f-b664-46355f0ab91f)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" podUID="7795d75a-dd56-486f-b664-46355f0ab91f"
Oct 30 22:45:19.187977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.187960    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"
Oct 30 22:45:19.188076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.187996    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww" event={"ID":"6379122a-c56a-4a2f-a170-597ac6eba2f1","Type":"ContainerDied","Data":"65314614b3b76fadf876104bc62bebdbe499aebc6b8db47715bdec82548e2d12"}
Oct 30 22:45:19.189603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.189573    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"
Oct 30 22:45:19.189960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.189936    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr" event={"ID":"261c1a76-4f45-49fa-822a-b211cefef3fa","Type":"ContainerDied","Data":"6a5456036faa153d2c07ce5a7d0838d5100df7f4e215c1b56a1fcc794fef65e0"}
Oct 30 22:45:19.194833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.194794    4882 generic.go:356] "Generic (PLEG): container finished" podID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerID="42c18caf0aa017d6aa5e830b23228ce10cd767a01ae8fc8c2eac67435dea3724" exitCode=60
Oct 30 22:45:19.194931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.194842    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerDied","Data":"42c18caf0aa017d6aa5e830b23228ce10cd767a01ae8fc8c2eac67435dea3724"}
Oct 30 22:45:19.195241 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.195222    4882 scope.go:122] "RemoveContainer" containerID="42c18caf0aa017d6aa5e830b23228ce10cd767a01ae8fc8c2eac67435dea3724"
Oct 30 22:45:19.195534 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:19.195512    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt_odh-ai-gateway-infra(780e1b40-53e0-4a25-9ef6-5ba366b12a6c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c"
Oct 30 22:45:19.199885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.199862    4882 scope.go:122] "RemoveContainer" containerID="848d27b3cf989038e571c32cf4efd77431894da8d8ccc72d8e33935ef7286a78"
Oct 30 22:45:19.211183 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.211160    4882 scope.go:122] "RemoveContainer" containerID="6edd0f963c4b12ef1529fc0aff87dd805b9d2f805c46afc9a3679a292f6e3e20"
Oct 30 22:45:19.221138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.221112    4882 scope.go:122] "RemoveContainer" containerID="501cda377c047cf1d49d5533fed6b52ee616522e177330db8be95b44351bfda2"
Oct 30 22:45:19.299600 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.299563    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:19.418553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.418515    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:19.430662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.430629    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-56f3fe-29890005-dcsww"]
Oct 30 22:45:19.544471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.541351    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:19.555564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.550654    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:19.555564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.555065    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-extmod-a-1a5f4a-29890005-dz8wr"]
Oct 30 22:45:19.613631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.612855    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:19.634077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.633991    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\" (UniqueName: \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\") pod \"5f957e98-5367-4753-bed1-9cea4014f17b\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") "
Oct 30 22:45:19.634142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.634109    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\") pod \"5f957e98-5367-4753-bed1-9cea4014f17b\" (UID: \"5f957e98-5367-4753-bed1-9cea4014f17b\") "
Oct 30 22:45:19.634744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.634717    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca" pod "5f957e98-5367-4753-bed1-9cea4014f17b" (UID: "5f957e98-5367-4753-bed1-9cea4014f17b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.637271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.637241    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7" pod "5f957e98-5367-4753-bed1-9cea4014f17b" (UID: "5f957e98-5367-4753-bed1-9cea4014f17b"). InnerVolumeSpecName "kube-api-access-qk2n7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.693742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.693713    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:19.735026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.734984    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\") pod \"91b6445b-cb6e-4b09-bf29-e8939defc5df\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") "
Oct 30 22:45:19.735226 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.735205    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\" (UniqueName: \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\") pod \"91b6445b-cb6e-4b09-bf29-e8939defc5df\" (UID: \"91b6445b-cb6e-4b09-bf29-e8939defc5df\") "
Oct 30 22:45:19.735330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.735307    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca" pod "91b6445b-cb6e-4b09-bf29-e8939defc5df" (UID: "91b6445b-cb6e-4b09-bf29-e8939defc5df"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:19.735662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.735643    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/91b6445b-cb6e-4b09-bf29-e8939defc5df-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.735745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.735666    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qk2n7\" (UniqueName: \"kubernetes.io/projected/5f957e98-5367-4753-bed1-9cea4014f17b-kube-api-access-qk2n7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.735745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.735680    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5f957e98-5367-4753-bed1-9cea4014f17b-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.737274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.737251    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b" pod "91b6445b-cb6e-4b09-bf29-e8939defc5df" (UID: "91b6445b-cb6e-4b09-bf29-e8939defc5df"). InnerVolumeSpecName "kube-api-access-crv2b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:19.836915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.836885    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crv2b\" (UniqueName: \"kubernetes.io/projected/91b6445b-cb6e-4b09-bf29-e8939defc5df-kube-api-access-crv2b\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:19.936535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:19.936496    4882 scope.go:122] "RemoveContainer" containerID="2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc"
Oct 30 22:45:20.205072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.205035    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerStarted","Data":"9105dfd4792cb5e239dc23a7de719162aa71102f8f983f88d144ef12601c66c4"}
Oct 30 22:45:20.206494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.206469    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"
Oct 30 22:45:20.206494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.206477    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s" event={"ID":"5f957e98-5367-4753-bed1-9cea4014f17b","Type":"ContainerDied","Data":"688975b6f18a1e35b3eee7ccf284f9469f8b00bb95f5d2f8b2313ec33bf5581a"}
Oct 30 22:45:20.206720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.206532    4882 scope.go:122] "RemoveContainer" containerID="1c40c6967f4fd41d739a71ae85cd96d2e2ecb7523fc55d4787ed866447ba3c39"
Oct 30 22:45:20.208657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.208628    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerStarted","Data":"08e76edc75b053c2b02f8989ed0dbf75ae45f93828a293f6952a5c2655cd001e"}
Oct 30 22:45:20.212486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.212460    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerStarted","Data":"178ce28acbde0c164a00fd3e811c8b60c0b56a4d405b8d2baca2f2ecf75d3e64"}
Oct 30 22:45:20.215620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.215600    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"
Oct 30 22:45:20.215735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.215629    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8" event={"ID":"91b6445b-cb6e-4b09-bf29-e8939defc5df","Type":"ContainerDied","Data":"6b2fce151ef0355efc5a6b5ed538cb391ec6b09c119ad6da0b2d35218e3a6b05"}
Oct 30 22:45:20.218683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.218561    4882 generic.go:356] "Generic (PLEG): container finished" podID="20cc509b-0284-458a-8735-482f710d7f62" containerID="27c99dd27081bd2865a7b1799370764430f243fb46df2f296636552e1ccf6af8" exitCode=60
Oct 30 22:45:20.218683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.218662    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerDied","Data":"27c99dd27081bd2865a7b1799370764430f243fb46df2f296636552e1ccf6af8"}
Oct 30 22:45:20.218910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.218889    4882 scope.go:122] "RemoveContainer" containerID="796662be1205ad6897fd28181bd7fc9ad9864f53fdd7b82aa907c10c46bdd146"
Oct 30 22:45:20.219026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.219007    4882 scope.go:122] "RemoveContainer" containerID="27c99dd27081bd2865a7b1799370764430f243fb46df2f296636552e1ccf6af8"
Oct 30 22:45:20.219316 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:20.219291    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8_odh-ai-gateway-infra(20cc509b-0284-458a-8735-482f710d7f62)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" podUID="20cc509b-0284-458a-8735-482f710d7f62"
Oct 30 22:45:20.220632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.220610    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerStarted","Data":"9c84991a82257fbca3a923754b905dff4142c9347fe0ee6f96cae601ffe4c9ea"}
Oct 30 22:45:20.237336 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.237308    4882 scope.go:122] "RemoveContainer" containerID="adeaa33f92b3e93a19a881125372b13ea5cc6fad24fe8038f924ff93362b6a7c"
Oct 30 22:45:20.313663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.313633    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:20.483961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.483933    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:20.547306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.547277    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\" (UniqueName: \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\") pod \"7795d75a-dd56-486f-b664-46355f0ab91f\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") "
Oct 30 22:45:20.547498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.547420    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\") pod \"7795d75a-dd56-486f-b664-46355f0ab91f\" (UID: \"7795d75a-dd56-486f-b664-46355f0ab91f\") "
Oct 30 22:45:20.547776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.547752    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca" pod "7795d75a-dd56-486f-b664-46355f0ab91f" (UID: "7795d75a-dd56-486f-b664-46355f0ab91f"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.547922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.547902    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7795d75a-dd56-486f-b664-46355f0ab91f-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.549422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.549396    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf" pod "7795d75a-dd56-486f-b664-46355f0ab91f" (UID: "7795d75a-dd56-486f-b664-46355f0ab91f"). InnerVolumeSpecName "kube-api-access-b2dkf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.649144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.649058    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2dkf\" (UniqueName: \"kubernetes.io/projected/7795d75a-dd56-486f-b664-46355f0ab91f-kube-api-access-b2dkf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.677003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.676965    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:20.811427 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:20.811386    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:45:20.820368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.820340    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:20.851973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.851930    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\" (UniqueName: \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\") pod \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") "
Oct 30 22:45:20.852132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.852033    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\") pod \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\" (UID: \"780e1b40-53e0-4a25-9ef6-5ba366b12a6c\") "
Oct 30 22:45:20.852447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.852425    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca" pod "780e1b40-53e0-4a25-9ef6-5ba366b12a6c" (UID: "780e1b40-53e0-4a25-9ef6-5ba366b12a6c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:20.854134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.854114    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6" pod "780e1b40-53e0-4a25-9ef6-5ba366b12a6c" (UID: "780e1b40-53e0-4a25-9ef6-5ba366b12a6c"). InnerVolumeSpecName "kube-api-access-kjqm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:20.945656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.945562    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" path="/var/lib/kubelet/pods/261c1a76-4f45-49fa-822a-b211cefef3fa/volumes"
Oct 30 22:45:20.946037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.946022    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" path="/var/lib/kubelet/pods/6379122a-c56a-4a2f-a170-597ac6eba2f1/volumes"
Oct 30 22:45:20.946441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.946429    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79229eb7-ee79-4b8a-8add-178dc1648b57" path="/var/lib/kubelet/pods/79229eb7-ee79-4b8a-8add-178dc1648b57/volumes"
Oct 30 22:45:20.953316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.953271    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqm6\" (UniqueName: \"kubernetes.io/projected/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-kube-api-access-kjqm6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:20.953316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:20.953301    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/780e1b40-53e0-4a25-9ef6-5ba366b12a6c-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:21.079483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.079450    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:21.083250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.083223    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29890005-tfw6s"]
Oct 30 22:45:21.193684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.193652    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:21.198066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.198000    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-5bb0b1-29890005-skvr8"]
Oct 30 22:45:21.231857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.231825    4882 generic.go:356] "Generic (PLEG): container finished" podID="1430c861-1090-42ff-a464-7bf32ad63e75" containerID="9105dfd4792cb5e239dc23a7de719162aa71102f8f983f88d144ef12601c66c4" exitCode=60
Oct 30 22:45:21.232292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.231893    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerDied","Data":"9105dfd4792cb5e239dc23a7de719162aa71102f8f983f88d144ef12601c66c4"}
Oct 30 22:45:21.232292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.231962    4882 scope.go:122] "RemoveContainer" containerID="f4ef3b7e5f381134200b4d45b9f7f3034668f8499519f4e736472acf8d1b5829"
Oct 30 22:45:21.232376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.232310    4882 scope.go:122] "RemoveContainer" containerID="9105dfd4792cb5e239dc23a7de719162aa71102f8f983f88d144ef12601c66c4"
Oct 30 22:45:21.232682 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:21.232660    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc_odh-ai-gateway-infra(1430c861-1090-42ff-a464-7bf32ad63e75)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" podUID="1430c861-1090-42ff-a464-7bf32ad63e75"
Oct 30 22:45:21.235088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.235067    4882 generic.go:356] "Generic (PLEG): container finished" podID="2a9340e4-b702-4077-8965-eb968fa24422" containerID="08e76edc75b053c2b02f8989ed0dbf75ae45f93828a293f6952a5c2655cd001e" exitCode=60
Oct 30 22:45:21.235185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.235140    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerDied","Data":"08e76edc75b053c2b02f8989ed0dbf75ae45f93828a293f6952a5c2655cd001e"}
Oct 30 22:45:21.235527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.235491    4882 scope.go:122] "RemoveContainer" containerID="08e76edc75b053c2b02f8989ed0dbf75ae45f93828a293f6952a5c2655cd001e"
Oct 30 22:45:21.235825 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:21.235805    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5_odh-ai-gateway-infra(2a9340e4-b702-4077-8965-eb968fa24422)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" podUID="2a9340e4-b702-4077-8965-eb968fa24422"
Oct 30 22:45:21.236562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.236546    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"
Oct 30 22:45:21.236562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.236552    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt" event={"ID":"780e1b40-53e0-4a25-9ef6-5ba366b12a6c","Type":"ContainerDied","Data":"5867a1021eed16f8c1bafb54ab63fc2557b9f311191be3d8e1dc8db03e7512fe"}
Oct 30 22:45:21.238998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.238978    4882 generic.go:356] "Generic (PLEG): container finished" podID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerID="178ce28acbde0c164a00fd3e811c8b60c0b56a4d405b8d2baca2f2ecf75d3e64" exitCode=60
Oct 30 22:45:21.239092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.239046    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerDied","Data":"178ce28acbde0c164a00fd3e811c8b60c0b56a4d405b8d2baca2f2ecf75d3e64"}
Oct 30 22:45:21.239404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.239386    4882 scope.go:122] "RemoveContainer" containerID="178ce28acbde0c164a00fd3e811c8b60c0b56a4d405b8d2baca2f2ecf75d3e64"
Oct 30 22:45:21.239684 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:21.239663    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg_odh-ai-gateway-infra(c59e7d31-e436-4408-b2dc-755f0b57d1ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed"
Oct 30 22:45:21.240528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.240509    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"
Oct 30 22:45:21.240528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.240515    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv" event={"ID":"7795d75a-dd56-486f-b664-46355f0ab91f","Type":"ContainerDied","Data":"ca7828d63b7f4a1fbd038b5948ab960f4ee9005df9e736944e360c8a6d1c01b1"}
Oct 30 22:45:21.243896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.243870    4882 generic.go:356] "Generic (PLEG): container finished" podID="1679375d-7196-4326-abdf-0735b3b6e814" containerID="b75aad304fa96d63ea6e0c40baf8ccc27c82e81fe13cdec205cb616796e7ace0" exitCode=60
Oct 30 22:45:21.243988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.243950    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerDied","Data":"b75aad304fa96d63ea6e0c40baf8ccc27c82e81fe13cdec205cb616796e7ace0"}
Oct 30 22:45:21.244297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.244279    4882 scope.go:122] "RemoveContainer" containerID="b75aad304fa96d63ea6e0c40baf8ccc27c82e81fe13cdec205cb616796e7ace0"
Oct 30 22:45:21.244529 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:21.244508    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp_odh-ai-gateway-infra(1679375d-7196-4326-abdf-0735b3b6e814)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" podUID="1679375d-7196-4326-abdf-0735b3b6e814"
Oct 30 22:45:21.247003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.246988    4882 scope.go:122] "RemoveContainer" containerID="6748de0dba8e0b4b8821b287d68068b73a4b9098028a6b57b7cf24de2356c79f"
Oct 30 22:45:21.247506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.247484    4882 generic.go:356] "Generic (PLEG): container finished" podID="c8fda827-f646-4860-b348-adb52b07ca42" containerID="9c84991a82257fbca3a923754b905dff4142c9347fe0ee6f96cae601ffe4c9ea" exitCode=60
Oct 30 22:45:21.247578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.247536    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerDied","Data":"9c84991a82257fbca3a923754b905dff4142c9347fe0ee6f96cae601ffe4c9ea"}
Oct 30 22:45:21.247870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.247857    4882 scope.go:122] "RemoveContainer" containerID="9c84991a82257fbca3a923754b905dff4142c9347fe0ee6f96cae601ffe4c9ea"
Oct 30 22:45:21.248078 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:21.248061    4882 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk_odh-ai-gateway-infra(c8fda827-f646-4860-b348-adb52b07ca42)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" podUID="c8fda827-f646-4860-b348-adb52b07ca42"
Oct 30 22:45:21.260685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.260666    4882 scope.go:122] "RemoveContainer" containerID="42c18caf0aa017d6aa5e830b23228ce10cd767a01ae8fc8c2eac67435dea3724"
Oct 30 22:45:21.270234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.270215    4882 scope.go:122] "RemoveContainer" containerID="88262dad0575815a826121c5693a5e275eb159eff241d2ba7ea65b41047b1cf2"
Oct 30 22:45:21.283748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.283726    4882 scope.go:122] "RemoveContainer" containerID="8342b7fcadeecab4fbd34b9900088d61cca019452cf3a057d566c1d4f22abaa3"
Oct 30 22:45:21.297229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.297205    4882 scope.go:122] "RemoveContainer" containerID="2403f033978144626087156ef044b76d036177d85d4de335416e4188231010fc"
Oct 30 22:45:21.314248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.314217    4882 scope.go:122] "RemoveContainer" containerID="7900bb8cb54d1041b391c592e0a1397fe388a71d292c39d95afcf37a3bbb0bda"
Oct 30 22:45:21.337604 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.337548    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:21.622628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.622574    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:21.663617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.663557    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:21.667350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.667320    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-5e884c-29890005-bk9nv"]
Oct 30 22:45:21.741027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.740990    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:21.859913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.859878    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:21.904224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.904192    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:21.910064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.910032    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c29159-29890005-kfwpt"]
Oct 30 22:45:21.933663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.933632    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:21.964136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.964100    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\") pod \"20cc509b-0284-458a-8735-482f710d7f62\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") "
Oct 30 22:45:21.964303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.964167    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\" (UniqueName: \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\") pod \"20cc509b-0284-458a-8735-482f710d7f62\" (UID: \"20cc509b-0284-458a-8735-482f710d7f62\") "
Oct 30 22:45:21.964462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.964440    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca" pod "20cc509b-0284-458a-8735-482f710d7f62" (UID: "20cc509b-0284-458a-8735-482f710d7f62"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:21.966289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.966267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck" pod "20cc509b-0284-458a-8735-482f710d7f62" (UID: "20cc509b-0284-458a-8735-482f710d7f62"). InnerVolumeSpecName "kube-api-access-qfzck". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:21.976084 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:21.976055    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:22.066178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.066136    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/20cc509b-0284-458a-8735-482f710d7f62-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.066178 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.066169    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfzck\" (UniqueName: \"kubernetes.io/projected/20cc509b-0284-458a-8735-482f710d7f62-kube-api-access-qfzck\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.264435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.264409    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"
Oct 30 22:45:22.265003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.264440    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8" event={"ID":"20cc509b-0284-458a-8735-482f710d7f62","Type":"ContainerDied","Data":"10033f325f56b7618702bac5bd781e524a6f4233b99acd0deb0e4039c7152ee8"}
Oct 30 22:45:22.265003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.264517    4882 scope.go:122] "RemoveContainer" containerID="27c99dd27081bd2865a7b1799370764430f243fb46df2f296636552e1ccf6af8"
Oct 30 22:45:22.467977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.467947    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:22.484383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.484343    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:22.492341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.492310    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-4d14e0-29890005-bqnd8"]
Oct 30 22:45:22.512096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.512070    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:22.530239 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.530216    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:22.534510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.534486    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:22.574880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.574843    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\") pod \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") "
Oct 30 22:45:22.575063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.574911    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\") pod \"2a9340e4-b702-4077-8965-eb968fa24422\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") "
Oct 30 22:45:22.575063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.574935    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\" (UniqueName: \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\") pod \"c8fda827-f646-4860-b348-adb52b07ca42\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") "
Oct 30 22:45:22.575063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.574982    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\") pod \"c8fda827-f646-4860-b348-adb52b07ca42\" (UID: \"c8fda827-f646-4860-b348-adb52b07ca42\") "
Oct 30 22:45:22.575063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575018    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\") pod \"1430c861-1090-42ff-a464-7bf32ad63e75\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") "
Oct 30 22:45:22.575341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575124    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\" (UniqueName: \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\") pod \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\" (UID: \"c59e7d31-e436-4408-b2dc-755f0b57d1ed\") "
Oct 30 22:45:22.575341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575162    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\" (UniqueName: \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\") pod \"1430c861-1090-42ff-a464-7bf32ad63e75\" (UID: \"1430c861-1090-42ff-a464-7bf32ad63e75\") "
Oct 30 22:45:22.575341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575196    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\" (UniqueName: \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\") pod \"2a9340e4-b702-4077-8965-eb968fa24422\" (UID: \"2a9340e4-b702-4077-8965-eb968fa24422\") "
Oct 30 22:45:22.575341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575241    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca" pod "c59e7d31-e436-4408-b2dc-755f0b57d1ed" (UID: "c59e7d31-e436-4408-b2dc-755f0b57d1ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.575601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575463    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca" pod "1430c861-1090-42ff-a464-7bf32ad63e75" (UID: "1430c861-1090-42ff-a464-7bf32ad63e75"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.575688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575566    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca" pod "2a9340e4-b702-4077-8965-eb968fa24422" (UID: "2a9340e4-b702-4077-8965-eb968fa24422"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.575766 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575712    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca" pod "c8fda827-f646-4860-b348-adb52b07ca42" (UID: "c8fda827-f646-4860-b348-adb52b07ca42"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.575876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575851    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2a9340e4-b702-4077-8965-eb968fa24422-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.575951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575882    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8fda827-f646-4860-b348-adb52b07ca42-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.575951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575898    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1430c861-1090-42ff-a464-7bf32ad63e75-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.575951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.575912    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c59e7d31-e436-4408-b2dc-755f0b57d1ed-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.577271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.577248    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj" pod "c8fda827-f646-4860-b348-adb52b07ca42" (UID: "c8fda827-f646-4860-b348-adb52b07ca42"). InnerVolumeSpecName "kube-api-access-pkgkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.577410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.577329    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr" pod "c59e7d31-e436-4408-b2dc-755f0b57d1ed" (UID: "c59e7d31-e436-4408-b2dc-755f0b57d1ed"). InnerVolumeSpecName "kube-api-access-9pfzr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.577495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.577476    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx" pod "2a9340e4-b702-4077-8965-eb968fa24422" (UID: "2a9340e4-b702-4077-8965-eb968fa24422"). InnerVolumeSpecName "kube-api-access-s9qmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.577538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.577501    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5" pod "1430c861-1090-42ff-a464-7bf32ad63e75" (UID: "1430c861-1090-42ff-a464-7bf32ad63e75"). InnerVolumeSpecName "kube-api-access-4xlb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:22.677262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.677219    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pfzr\" (UniqueName: \"kubernetes.io/projected/c59e7d31-e436-4408-b2dc-755f0b57d1ed-kube-api-access-9pfzr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.677262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.677254    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4xlb5\" (UniqueName: \"kubernetes.io/projected/1430c861-1090-42ff-a464-7bf32ad63e75-kube-api-access-4xlb5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.677262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.677269    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9qmx\" (UniqueName: \"kubernetes.io/projected/2a9340e4-b702-4077-8965-eb968fa24422-kube-api-access-s9qmx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.677524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.677284    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkgkj\" (UniqueName: \"kubernetes.io/projected/c8fda827-f646-4860-b348-adb52b07ca42-kube-api-access-pkgkj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:22.788912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.788826    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:22.933844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.933818    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:22.944128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.944099    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20cc509b-0284-458a-8735-482f710d7f62" path="/var/lib/kubelet/pods/20cc509b-0284-458a-8735-482f710d7f62/volumes"
Oct 30 22:45:22.944596 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.944571    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" path="/var/lib/kubelet/pods/5f957e98-5367-4753-bed1-9cea4014f17b/volumes"
Oct 30 22:45:22.944995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.944982    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" path="/var/lib/kubelet/pods/7795d75a-dd56-486f-b664-46355f0ab91f/volumes"
Oct 30 22:45:22.945680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.945665    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" path="/var/lib/kubelet/pods/780e1b40-53e0-4a25-9ef6-5ba366b12a6c/volumes"
Oct 30 22:45:22.946068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.946053    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" path="/var/lib/kubelet/pods/91b6445b-cb6e-4b09-bf29-e8939defc5df/volumes"
Oct 30 22:45:22.980998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.980967    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\") pod \"1679375d-7196-4326-abdf-0735b3b6e814\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") "
Oct 30 22:45:22.981179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.981102    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\" (UniqueName: \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\") pod \"1679375d-7196-4326-abdf-0735b3b6e814\" (UID: \"1679375d-7196-4326-abdf-0735b3b6e814\") "
Oct 30 22:45:22.981432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.981400    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca" pod "1679375d-7196-4326-abdf-0735b3b6e814" (UID: "1679375d-7196-4326-abdf-0735b3b6e814"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:22.983199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:22.983176    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x" pod "1679375d-7196-4326-abdf-0735b3b6e814" (UID: "1679375d-7196-4326-abdf-0735b3b6e814"). InnerVolumeSpecName "kube-api-access-z2q7x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:23.083199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.083161    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2q7x\" (UniqueName: \"kubernetes.io/projected/1679375d-7196-4326-abdf-0735b3b6e814-kube-api-access-z2q7x\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:23.083362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.083202    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/1679375d-7196-4326-abdf-0735b3b6e814-maas-api-service-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:23.280624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.280575    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"
Oct 30 22:45:23.281132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.280624    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5" event={"ID":"2a9340e4-b702-4077-8965-eb968fa24422","Type":"ContainerDied","Data":"af3fb2b74e1a833b63961172470e4a6a1c07d33da9037f113e092130c59d174f"}
Oct 30 22:45:23.281132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.280688    4882 scope.go:122] "RemoveContainer" containerID="08e76edc75b053c2b02f8989ed0dbf75ae45f93828a293f6952a5c2655cd001e"
Oct 30 22:45:23.282281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.282262    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"
Oct 30 22:45:23.282408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.282303    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg" event={"ID":"c59e7d31-e436-4408-b2dc-755f0b57d1ed","Type":"ContainerDied","Data":"fc7aabbc3abd56f9ee666b14f311d06e35fea0b9f35f3a780286d187ac835003"}
Oct 30 22:45:23.284199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.284182    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"
Oct 30 22:45:23.284341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.284231    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp" event={"ID":"1679375d-7196-4326-abdf-0735b3b6e814","Type":"ContainerDied","Data":"34f3b877890f3b33f56275776554fab5d935442cff5aad09f5775b7d84fbc9c1"}
Oct 30 22:45:23.287651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.287625    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk" event={"ID":"c8fda827-f646-4860-b348-adb52b07ca42","Type":"ContainerDied","Data":"7479302b2e028fc560a5b8f409a0f2f8c5c3db22a7340b143c4150f7158a364b"}
Oct 30 22:45:23.287651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.287645    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"
Oct 30 22:45:23.289267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.289243    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc" event={"ID":"1430c861-1090-42ff-a464-7bf32ad63e75","Type":"ContainerDied","Data":"d3cf87dd67ee5355ada78921aedff5e03366df4aa51bd842bfe383f33e6551e9"}
Oct 30 22:45:23.289267 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.289262    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"
Oct 30 22:45:23.293167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.293144    4882 scope.go:122] "RemoveContainer" containerID="178ce28acbde0c164a00fd3e811c8b60c0b56a4d405b8d2baca2f2ecf75d3e64"
Oct 30 22:45:23.304942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.304922    4882 scope.go:122] "RemoveContainer" containerID="b75aad304fa96d63ea6e0c40baf8ccc27c82e81fe13cdec205cb616796e7ace0"
Oct 30 22:45:23.317686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.317658    4882 scope.go:122] "RemoveContainer" containerID="9c84991a82257fbca3a923754b905dff4142c9347fe0ee6f96cae601ffe4c9ea"
Oct 30 22:45:23.328858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.328831    4882 scope.go:122] "RemoveContainer" containerID="9105dfd4792cb5e239dc23a7de719162aa71102f8f983f88d144ef12601c66c4"
Oct 30 22:45:23.650509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.650426    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:23.656212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.656180    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-6fc60c-29890005-gwtcp"]
Oct 30 22:45:23.765751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.765716    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:23.771731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.771694    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-9a5623-29890005-dfkt5"]
Oct 30 22:45:23.882161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.882122    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:23.888544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.888505    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w1-323082-29890005-rktrg"]
Oct 30 22:45:23.997772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:23.997684    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:24.004445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.004414    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-ae7a1f-29890005-zmdlk"]
Oct 30 22:45:24.115038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.114994    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:24.120732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.120702    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-3c4802-29890005-cg6mc"]
Oct 30 22:45:24.943960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.943922    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" path="/var/lib/kubelet/pods/1430c861-1090-42ff-a464-7bf32ad63e75/volumes"
Oct 30 22:45:24.944447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.944428    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1679375d-7196-4326-abdf-0735b3b6e814" path="/var/lib/kubelet/pods/1679375d-7196-4326-abdf-0735b3b6e814/volumes"
Oct 30 22:45:24.944896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.944880    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a9340e4-b702-4077-8965-eb968fa24422" path="/var/lib/kubelet/pods/2a9340e4-b702-4077-8965-eb968fa24422/volumes"
Oct 30 22:45:24.945818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.945792    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" path="/var/lib/kubelet/pods/c59e7d31-e436-4408-b2dc-755f0b57d1ed/volumes"
Oct 30 22:45:24.946320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:24.946301    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c8fda827-f646-4860-b348-adb52b07ca42" path="/var/lib/kubelet/pods/c8fda827-f646-4860-b348-adb52b07ca42/volumes"
Oct 30 22:45:43.052423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:43.052339    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"]
Oct 30 22:45:43.102803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:43.102768    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3a4b4289-rtvpl"]
Oct 30 22:45:44.944617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:44.944549    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93f04dbb-9074-4ae8-8da1-41d7be82b0c3" path="/var/lib/kubelet/pods/93f04dbb-9074-4ae8-8da1-41d7be82b0c3/volumes"
Oct 30 22:45:50.808145 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:45:50.808107    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:45:50.853201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:50.853164    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:45:54.055142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:54.055103    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"]
Oct 30 22:45:54.102144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:54.102108    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8f1d2aa-4244g"]
Oct 30 22:45:54.969985 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:54.969937    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="933df945-d492-4035-a607-ad6e72926d9a" path="/var/lib/kubelet/pods/933df945-d492-4035-a607-ad6e72926d9a/volumes"
Oct 30 22:45:55.003104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.003068    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"]
Oct 30 22:45:55.004382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004351    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.004557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004544    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.004689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004676    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.004775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004764    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.004864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004854    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.004943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.004932    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.005022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005013    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.005092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005082    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.005173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005165    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.005243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005233    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.005326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005317    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.005395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005386    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.005479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005466    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.005554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005544    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.005651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005641    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.005734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005725    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.005835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005826    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.005904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005896    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.006005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.005996    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.006082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006073    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.006162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006154    4882 cpu_manager.go:414] "containerMap: removing container" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.006230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006221    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.006305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006297    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.006372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006363    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.006438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006430    4882 cpu_manager.go:414] "containerMap: removing container" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.006505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006496    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.006599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006575    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.006691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006680    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.006775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006767    4882 cpu_manager.go:414] "containerMap: removing container" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.006845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006835    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.006916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006908    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.006980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.006972    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.007059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007051    4882 cpu_manager.go:414] "containerMap: removing container" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.007130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007122    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.007203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007195    4882 cpu_manager.go:414] "containerMap: removing container" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.007276 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007267    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.007350 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007341    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.007419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007409    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.007491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007483    4882 cpu_manager.go:414] "containerMap: removing container" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.007557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007549    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.007650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007625    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.007650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007641    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007653    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007660    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007673    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007679    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007693    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007700    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007716    4882 cpu_manager.go:414] "containerMap: removing container" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007723    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007739    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007746    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007757    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007764    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007776    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.007791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007782    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007801    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007808    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007817    4882 cpu_manager.go:414] "containerMap: removing container" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007824    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007842    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007849    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007865    4882 cpu_manager.go:414] "containerMap: removing container" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007872    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007889    4882 cpu_manager.go:414] "containerMap: removing container" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.007896    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008070    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008083    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008094    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008103    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008116    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008124    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008133    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008140    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008148    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008156    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008164    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008172    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008182    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008192    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008200    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008210    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008221    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008231    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008240    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008248    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008256    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008267    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008277    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008290    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008299    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008308    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008317    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1aeee42b-cc59-4f1d-a55b-829ba217cb0d" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008327    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008336    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008348    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="780e1b40-53e0-4a25-9ef6-5ba366b12a6c" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008357    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="7795d75a-dd56-486f-b664-46355f0ab91f" containerName="cleanup"
Oct 30 22:45:55.008504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.008366    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1679375d-7196-4326-abdf-0735b3b6e814" containerName="cleanup"
Oct 30 22:45:55.014265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.014241    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.072700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.072665    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"]
Oct 30 22:45:55.082746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.082702    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.082962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.082789    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.082962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.082839    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9mn9m\" (UniqueName: \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.106387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.106359    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:45:55.106387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.106365    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:45:55.183898 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.183862    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9mn9m\" (UniqueName: \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.184105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.183982    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.184105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.184057    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.184812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.184788    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.186508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.186482    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.251840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.251758    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9mn9m\" (UniqueName: \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\") pod \"maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.349901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.349849    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:55.754437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:55.754400    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"]
Oct 30 22:45:56.565826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:56.565727    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp" event={"ID":"4b465a47-8f74-4a0b-8da4-bec0f0185cdf","Type":"ContainerStarted","Data":"c428ce29530cda277451ef560741fb49be1a66d2a6400ffed35579c1f5fddfe8"}
Oct 30 22:45:56.566294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:56.565836    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp" event={"ID":"4b465a47-8f74-4a0b-8da4-bec0f0185cdf","Type":"ContainerStarted","Data":"03c9385171b0debb394845714fae859ef87a947f490c387b0c601f9fc80958a7"}
Oct 30 22:45:56.685697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:56.685641    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp" podStartSLOduration=2.6856271830000003 podStartE2EDuration="2.685627183s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:45:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:45:56.683436296 +0000 UTC m=+4079.856760722" watchObservedRunningTime="2026-10-30 22:45:56.685627183 +0000 UTC m=+4079.858951609"
Oct 30 22:45:57.575947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:57.575905    4882 generic.go:356] "Generic (PLEG): container finished" podID="4b465a47-8f74-4a0b-8da4-bec0f0185cdf" containerID="c428ce29530cda277451ef560741fb49be1a66d2a6400ffed35579c1f5fddfe8" exitCode=0
Oct 30 22:45:57.576381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:57.575972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp" event={"ID":"4b465a47-8f74-4a0b-8da4-bec0f0185cdf","Type":"ContainerDied","Data":"c428ce29530cda277451ef560741fb49be1a66d2a6400ffed35579c1f5fddfe8"}
Oct 30 22:45:58.760067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.760025    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:58.824876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.824842    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\" (UniqueName: \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\") pod \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") "
Oct 30 22:45:58.825085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.824888    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\") pod \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") "
Oct 30 22:45:58.825085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.824934    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\") pod \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\" (UID: \"4b465a47-8f74-4a0b-8da4-bec0f0185cdf\") "
Oct 30 22:45:58.825355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.825331    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca" pod "4b465a47-8f74-4a0b-8da4-bec0f0185cdf" (UID: "4b465a47-8f74-4a0b-8da4-bec0f0185cdf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:45:58.825470 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.825454    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.827324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.827295    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m" pod "4b465a47-8f74-4a0b-8da4-bec0f0185cdf" (UID: "4b465a47-8f74-4a0b-8da4-bec0f0185cdf"). InnerVolumeSpecName "kube-api-access-9mn9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:45:58.827468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.827417    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca" pod "4b465a47-8f74-4a0b-8da4-bec0f0185cdf" (UID: "4b465a47-8f74-4a0b-8da4-bec0f0185cdf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:45:58.926381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.926345    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9mn9m\" (UniqueName: \"kubernetes.io/projected/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-kube-api-access-9mn9m\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:58.926381 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:58.926379    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4b465a47-8f74-4a0b-8da4-bec0f0185cdf-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:45:59.592950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:59.592921    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"
Oct 30 22:45:59.593132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:59.592953    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp" event={"ID":"4b465a47-8f74-4a0b-8da4-bec0f0185cdf","Type":"ContainerDied","Data":"03c9385171b0debb394845714fae859ef87a947f490c387b0c601f9fc80958a7"}
Oct 30 22:45:59.593132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:45:59.593001    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03c9385171b0debb394845714fae859ef87a947f490c387b0c601f9fc80958a7"
Oct 30 22:46:01.610129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.610039    4882 generic.go:356] "Generic (PLEG): container finished" podID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerID="57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea" exitCode=0
Oct 30 22:46:01.610129 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.610108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerDied","Data":"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"}
Oct 30 22:46:01.610688 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.610449    4882 scope.go:122] "RemoveContainer" containerID="57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"
Oct 30 22:46:01.612036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.612008    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:01.612147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.612078    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:01.612147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:01.612110    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:02.349619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:02.349008    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:46:02.619140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:02.619055    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerStarted","Data":"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"}
Oct 30 22:46:03.008939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:03.008853    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:46:03.010128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:03.010101    4882 patch_prober.go:29] interesting pod/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 22:46:03.010269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:03.010150    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 22:46:04.008831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:04.008792    4882 patch_prober.go:29] interesting pod/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 22:46:04.009221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:04.008858    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 22:46:05.008492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:05.008458    4882 patch_prober.go:29] interesting pod/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 22:46:05.008700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:05.008511    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 22:46:06.008682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:06.008642    4882 patch_prober.go:29] interesting pod/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 22:46:06.009095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:06.008724    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 22:46:06.600780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:06.600744    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:46:06.703962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:06.703922    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:46:07.008863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:07.008769    4882 patch_prober.go:29] interesting pod/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused" start-of-body=
Oct 30 22:46:07.008863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:07.008847    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.52:15021/healthz/ready\": dial tcp 10.132.1.52:15021: connect: connection refused"
Oct 30 22:46:08.014050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:08.014017    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:46:08.014505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:08.014491    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:46:08.015262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:08.015240    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:46:11.602070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:11.602022    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" podUID="165fd979-8b88-482e-89a4-c27e621f1a42" containerName="payload-processing" containerID="cri-o://79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c" gracePeriod=25
Oct 30 22:46:11.704842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:11.704791    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" podUID="aa067076-f410-4aeb-90de-0251ba09a294" containerName="payload-pre-processing" containerID="cri-o://805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca" gracePeriod=25
Oct 30 22:46:11.885663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:11.885632    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:46:11.994504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:11.994477    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:46:11.999071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:11.999046    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"]
Oct 30 22:46:12.003442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.003416    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-be6ecc-crrcp"]
Oct 30 22:46:12.013279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013243    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\") pod \"165fd979-8b88-482e-89a4-c27e621f1a42\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") "
Oct 30 22:46:12.013279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013282    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\" (UniqueName: \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\") pod \"165fd979-8b88-482e-89a4-c27e621f1a42\" (UID: \"165fd979-8b88-482e-89a4-c27e621f1a42\") "
Oct 30 22:46:12.013495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013343    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\") pod \"aa067076-f410-4aeb-90de-0251ba09a294\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") "
Oct 30 22:46:12.013495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013447    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\" (UniqueName: \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\") pod \"aa067076-f410-4aeb-90de-0251ba09a294\" (UID: \"aa067076-f410-4aeb-90de-0251ba09a294\") "
Oct 30 22:46:12.013842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013812    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume" pod "165fd979-8b88-482e-89a4-c27e621f1a42" (UID: "165fd979-8b88-482e-89a4-c27e621f1a42"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:12.013931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013867    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume" pod "aa067076-f410-4aeb-90de-0251ba09a294" (UID: "aa067076-f410-4aeb-90de-0251ba09a294"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:12.013980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.013964    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/165fd979-8b88-482e-89a4-c27e621f1a42-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.015650 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.015625    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8" pod "165fd979-8b88-482e-89a4-c27e621f1a42" (UID: "165fd979-8b88-482e-89a4-c27e621f1a42"). InnerVolumeSpecName "kube-api-access-pw7w8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:12.015748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.015663    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb" pod "aa067076-f410-4aeb-90de-0251ba09a294" (UID: "aa067076-f410-4aeb-90de-0251ba09a294"). InnerVolumeSpecName "kube-api-access-jhpwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:12.115199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.115101    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pw7w8\" (UniqueName: \"kubernetes.io/projected/165fd979-8b88-482e-89a4-c27e621f1a42-kube-api-access-pw7w8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.115199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.115136    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa067076-f410-4aeb-90de-0251ba09a294-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.115199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.115151    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhpwb\" (UniqueName: \"kubernetes.io/projected/aa067076-f410-4aeb-90de-0251ba09a294-kube-api-access-jhpwb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:12.714964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.714923    4882 generic.go:356] "Generic (PLEG): container finished" podID="165fd979-8b88-482e-89a4-c27e621f1a42" containerID="79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c" exitCode=0
Oct 30 22:46:12.715443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.714971    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"
Oct 30 22:46:12.715443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.715013    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" event={"ID":"165fd979-8b88-482e-89a4-c27e621f1a42","Type":"ContainerDied","Data":"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"}
Oct 30 22:46:12.715443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.715077    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4" event={"ID":"165fd979-8b88-482e-89a4-c27e621f1a42","Type":"ContainerDied","Data":"461831d9dc1c3e5c2cbbb4959d1a6e05d9534101bca94d92f813d94f6f3d22f2"}
Oct 30 22:46:12.715443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.715107    4882 scope.go:122] "RemoveContainer" containerID="79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"
Oct 30 22:46:12.716693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.716672    4882 generic.go:356] "Generic (PLEG): container finished" podID="aa067076-f410-4aeb-90de-0251ba09a294" containerID="805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca" exitCode=0
Oct 30 22:46:12.716789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.716739    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"
Oct 30 22:46:12.716789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.716751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" event={"ID":"aa067076-f410-4aeb-90de-0251ba09a294","Type":"ContainerDied","Data":"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"}
Oct 30 22:46:12.716873 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.716818    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx" event={"ID":"aa067076-f410-4aeb-90de-0251ba09a294","Type":"ContainerDied","Data":"8d436c78b3a73ea36f9ba7963aff3cb7e1cc8b2e7d4ec3db3b7529c7ca0be094"}
Oct 30 22:46:12.733690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.733668    4882 scope.go:122] "RemoveContainer" containerID="79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"
Oct 30 22:46:12.734000 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:46:12.733981    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c\": container with ID starting with 79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c not found: ID does not exist" containerID="79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"
Oct 30 22:46:12.734061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.734009    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c"} err="failed to get container status \"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c\": rpc error: code = NotFound desc = could not find container \"79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c\": container with ID starting with 79082208d5b10230392e404a6d312b0325c98f4b03753ddf159983c05305887c not found: ID does not exist"
Oct 30 22:46:12.734061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.734027    4882 scope.go:122] "RemoveContainer" containerID="805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"
Oct 30 22:46:12.743880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.743860    4882 scope.go:122] "RemoveContainer" containerID="805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"
Oct 30 22:46:12.744166 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:46:12.744143    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca\": container with ID starting with 805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca not found: ID does not exist" containerID="805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"
Oct 30 22:46:12.744215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.744173    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca"} err="failed to get container status \"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca\": rpc error: code = NotFound desc = could not find container \"805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca\": container with ID starting with 805d0e73d6b00aa9d16c4dd0737a324023c04f2ec9ff0d43ab9af79603c88eca not found: ID does not exist"
Oct 30 22:46:12.884058 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.883965    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:46:12.934631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.934579    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-be6ecc-75c5d958f5-8lgz4"]
Oct 30 22:46:12.945880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.944852    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="165fd979-8b88-482e-89a4-c27e621f1a42" path="/var/lib/kubelet/pods/165fd979-8b88-482e-89a4-c27e621f1a42/volumes"
Oct 30 22:46:12.945880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.945478    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b465a47-8f74-4a0b-8da4-bec0f0185cdf" path="/var/lib/kubelet/pods/4b465a47-8f74-4a0b-8da4-bec0f0185cdf/volumes"
Oct 30 22:46:12.999597 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:12.999561    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:46:13.004380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:13.004348    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-be6ecc-67d458fb99-bgwzx"]
Oct 30 22:46:14.943523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:14.943489    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa067076-f410-4aeb-90de-0251ba09a294" path="/var/lib/kubelet/pods/aa067076-f410-4aeb-90de-0251ba09a294/volumes"
Oct 30 22:46:20.808567 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:46:20.808521    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:46:25.053547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:25.053500    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"]
Oct 30 22:46:25.102085 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:25.102045    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c61259b9-4pz6q"]
Oct 30 22:46:26.193855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.193812    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:46:26.194746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194722    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4b465a47-8f74-4a0b-8da4-bec0f0185cdf" containerName="revoke-keys"
Oct 30 22:46:26.194746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194745    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b465a47-8f74-4a0b-8da4-bec0f0185cdf" containerName="revoke-keys"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194771    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194778    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c8fda827-f646-4860-b348-adb52b07ca42" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194791    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194798    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194815    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194822    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194837    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194844    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a9340e4-b702-4077-8965-eb968fa24422" containerName="cleanup"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194875    4882 cpu_manager.go:414] "containerMap: removing container" podUID="aa067076-f410-4aeb-90de-0251ba09a294" containerName="payload-pre-processing"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194882    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa067076-f410-4aeb-90de-0251ba09a294" containerName="payload-pre-processing"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194907    4882 cpu_manager.go:414] "containerMap: removing container" podUID="165fd979-8b88-482e-89a4-c27e621f1a42" containerName="payload-processing"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194915    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="165fd979-8b88-482e-89a4-c27e621f1a42" containerName="payload-processing"
Oct 30 22:46:26.194932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194935    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194942    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194954    4882 cpu_manager.go:414] "containerMap: removing container" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.194961    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195123    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa067076-f410-4aeb-90de-0251ba09a294" containerName="payload-pre-processing"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195138    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="165fd979-8b88-482e-89a4-c27e621f1a42" containerName="payload-processing"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195156    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c1a76-4f45-49fa-822a-b211cefef3fa" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195168    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="20cc509b-0284-458a-8735-482f710d7f62" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195179    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b465a47-8f74-4a0b-8da4-bec0f0185cdf" containerName="revoke-keys"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195191    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="91b6445b-cb6e-4b09-bf29-e8939defc5df" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195200    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c59e7d31-e436-4408-b2dc-755f0b57d1ed" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195210    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1430c861-1090-42ff-a464-7bf32ad63e75" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195222    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6379122a-c56a-4a2f-a170-597ac6eba2f1" containerName="cleanup"
Oct 30 22:46:26.195617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.195231    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f957e98-5367-4753-bed1-9cea4014f17b" containerName="cleanup"
Oct 30 22:46:26.200863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.200838    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.254122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.254085    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-6e0b99-openshift-default-dockercfg-pcv5l\""
Oct 30 22:46:26.279735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279697    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.279735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279736    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.279957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279821    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.279957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279853    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.279957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279880    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.279957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279914    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7n9m\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.280168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.279964    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.280168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.280075    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.280288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.280201    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.308222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.308189    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:46:26.381636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381580    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.381850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381659    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.381850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381703    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.381850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381742    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7n9m\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.381850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381774    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.381890    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382061    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382064    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382177 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382136    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382187    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382196    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382336    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.382791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.382772    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.384335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.384311    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.384475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.384451    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.444225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.444143    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.444407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.444266    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7n9m\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\") pod \"e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") " pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.515313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.515280    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:26.945037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.945005    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f6e8905-af5d-4972-9fcd-6b5dff8af2f2" path="/var/lib/kubelet/pods/6f6e8905-af5d-4972-9fcd-6b5dff8af2f2/volumes"
Oct 30 22:46:26.946744 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:46:26.946709    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71a90030_8be8_44df_a758_a6e886b7123b.slice/crio-9865f193450a2f0dfe364fac99a23671a0adca6798ae25f022e2fc13cc02a63b WatchSource:0}: Error finding container 9865f193450a2f0dfe364fac99a23671a0adca6798ae25f022e2fc13cc02a63b: Status 404 returned error can't find the container with id 9865f193450a2f0dfe364fac99a23671a0adca6798ae25f022e2fc13cc02a63b
Oct 30 22:46:26.949934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.949896    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:26.950033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.949980    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:26.950033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.950025    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:46:26.950118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:26.950030    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:46:27.825862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:27.825823    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" event={"ID":"71a90030-8be8-44df-a758-a6e886b7123b","Type":"ContainerStarted","Data":"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"}
Oct 30 22:46:27.826537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:27.826496    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" event={"ID":"71a90030-8be8-44df-a758-a6e886b7123b","Type":"ContainerStarted","Data":"9865f193450a2f0dfe364fac99a23671a0adca6798ae25f022e2fc13cc02a63b"}
Oct 30 22:46:28.013187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:28.013130    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podStartSLOduration=2.01311481 podStartE2EDuration="2.01311481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:28.010996228 +0000 UTC m=+4111.184320655" watchObservedRunningTime="2026-10-30 22:46:28.01311481 +0000 UTC m=+4111.186439237"
Oct 30 22:46:28.516493 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:28.516459    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:28.518424 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:28.518397    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 22:46:28.518616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:28.518453    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 22:46:29.516798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:29.516763    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 22:46:29.517249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:29.516821    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 22:46:30.515731 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:30.515695    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 22:46:30.515930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:30.515749    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 22:46:31.515779 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.515743    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 22:46:31.516208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.515802    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 22:46:31.651533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.651495    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"]
Oct 30 22:46:31.656418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.656389    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.709865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.709831    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"]
Oct 30 22:46:31.756071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.756021    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:46:31.756430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.756380    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:31.779005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.778924    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.779214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.779196    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.779320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.779310    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6ch8\" (UniqueName: \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.880931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.880893    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.881123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.880959    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.881123 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.880991    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6ch8\" (UniqueName: \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.882057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.882038    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.883414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.883397    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.952765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.952728    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6ch8\" (UniqueName: \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\") pod \"maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:31.970942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:31.970915    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:32.390166 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:46:32.390123    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6835e137_f50d_42bd_82c9_2e037a59e63f.slice/crio-65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3 WatchSource:0}: Error finding container 65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3: Status 404 returned error can't find the container with id 65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3
Oct 30 22:46:32.393667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:32.393645    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"]
Oct 30 22:46:32.516653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:32.516297    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused" start-of-body=
Oct 30 22:46:32.516653 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:32.516347    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.75:15021/healthz/ready\": dial tcp 10.132.1.75:15021: connect: connection refused"
Oct 30 22:46:32.884171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:32.884137    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74" event={"ID":"6835e137-f50d-42bd-82c9-2e037a59e63f","Type":"ContainerStarted","Data":"65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3"}
Oct 30 22:46:33.521135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:33.521103    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:33.521561 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:33.521544    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:33.522540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:33.522522    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:46:33.893026 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:33.892994    4882 generic.go:356] "Generic (PLEG): container finished" podID="6835e137-f50d-42bd-82c9-2e037a59e63f" containerID="af7406cd2092a5c6a6d48b819db1efe78e43b5e099eb04fea9ea7a935174d796" exitCode=0
Oct 30 22:46:33.893244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:33.893083    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74" event={"ID":"6835e137-f50d-42bd-82c9-2e037a59e63f","Type":"ContainerDied","Data":"af7406cd2092a5c6a6d48b819db1efe78e43b5e099eb04fea9ea7a935174d796"}
Oct 30 22:46:35.056850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.056819    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"]
Oct 30 22:46:35.058662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.058642    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:35.106610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.106549    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3f1f2dc2-l56v9"]
Oct 30 22:46:35.115082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.115053    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\" (UniqueName: \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\") pod \"6835e137-f50d-42bd-82c9-2e037a59e63f\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") "
Oct 30 22:46:35.115213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.115095    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\") pod \"6835e137-f50d-42bd-82c9-2e037a59e63f\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") "
Oct 30 22:46:35.115213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.115141    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\") pod \"6835e137-f50d-42bd-82c9-2e037a59e63f\" (UID: \"6835e137-f50d-42bd-82c9-2e037a59e63f\") "
Oct 30 22:46:35.115572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.115549    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca" pod "6835e137-f50d-42bd-82c9-2e037a59e63f" (UID: "6835e137-f50d-42bd-82c9-2e037a59e63f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:35.117297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.117276    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca" pod "6835e137-f50d-42bd-82c9-2e037a59e63f" (UID: "6835e137-f50d-42bd-82c9-2e037a59e63f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:35.117380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.117353    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8" pod "6835e137-f50d-42bd-82c9-2e037a59e63f" (UID: "6835e137-f50d-42bd-82c9-2e037a59e63f"). InnerVolumeSpecName "kube-api-access-q6ch8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:35.216715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.216627    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6ch8\" (UniqueName: \"kubernetes.io/projected/6835e137-f50d-42bd-82c9-2e037a59e63f-kube-api-access-q6ch8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:35.216715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.216664    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:35.216715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.216674    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6835e137-f50d-42bd-82c9-2e037a59e63f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:35.910433 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.910401    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"
Oct 30 22:46:35.910677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.910401    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74" event={"ID":"6835e137-f50d-42bd-82c9-2e037a59e63f","Type":"ContainerDied","Data":"65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3"}
Oct 30 22:46:35.910677 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:35.910569    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="65462a5830f943c32e37855a498c05a05d5cb5d7bdc798581d919b8053f207f3"
Oct 30 22:46:36.944299 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:36.944261    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9220c2f3-658c-47e8-ad87-f1bae051dd19" path="/var/lib/kubelet/pods/9220c2f3-658c-47e8-ad87-f1bae051dd19/volumes"
Oct 30 22:46:43.807464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:43.807432    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:46:43.917943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:43.917906    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:46:48.568986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.568950    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"]
Oct 30 22:46:48.569694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.569678    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6835e137-f50d-42bd-82c9-2e037a59e63f" containerName="revoke-keys"
Oct 30 22:46:48.569771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.569695    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6835e137-f50d-42bd-82c9-2e037a59e63f" containerName="revoke-keys"
Oct 30 22:46:48.569838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.569826    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6835e137-f50d-42bd-82c9-2e037a59e63f" containerName="revoke-keys"
Oct 30 22:46:48.574160 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.574139    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.628662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.628624    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:48.631479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.631438    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"]
Oct 30 22:46:48.675373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.675338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.675563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.675474    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb2jd\" (UniqueName: \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.675563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.675552    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.677523 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.677501    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:46:48.777126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.777092    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb2jd\" (UniqueName: \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.777338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.777164    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.777338 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.777305    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.778025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.778002    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.779614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.779577    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.808872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.808833    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing" containerID="cri-o://625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e" gracePeriod=25
Oct 30 22:46:48.839171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.839142    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb2jd\" (UniqueName: \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\") pod \"maas-api-revoke-keys-e2e-autores-5a5534-x76mp\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.888988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.888951    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:48.922945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:48.922894    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing" containerID="cri-o://4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1" gracePeriod=25
Oct 30 22:46:49.268759 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:46:49.268728    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaf67d68_ab4e_4d8f_bd07_3e8c6fb4f184.slice/crio-53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25 WatchSource:0}: Error finding container 53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25: Status 404 returned error can't find the container with id 53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25
Oct 30 22:46:49.271288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:49.271263    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"]
Oct 30 22:46:50.015974 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:50.015937    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp" event={"ID":"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184","Type":"ContainerStarted","Data":"cc36d87e45c24afe4caa4aa115953ff14c95cca40b9e8bb49493477787c1a555"}
Oct 30 22:46:50.016476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:50.015994    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp" event={"ID":"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184","Type":"ContainerStarted","Data":"53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25"}
Oct 30 22:46:50.135146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:50.135096    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp" podStartSLOduration=2.135081266 podStartE2EDuration="2.135081266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:50.132093041 +0000 UTC m=+4133.305417467" watchObservedRunningTime="2026-10-30 22:46:50.135081266 +0000 UTC m=+4133.308405692"
Oct 30 22:46:50.809673 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:46:50.809632    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:46:51.035444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:51.035407    4882 generic.go:356] "Generic (PLEG): container finished" podID="daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" containerID="cc36d87e45c24afe4caa4aa115953ff14c95cca40b9e8bb49493477787c1a555" exitCode=0
Oct 30 22:46:51.035969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:51.035489    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp" event={"ID":"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184","Type":"ContainerDied","Data":"cc36d87e45c24afe4caa4aa115953ff14c95cca40b9e8bb49493477787c1a555"}
Oct 30 22:46:52.201823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.201788    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:52.322888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.322803    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\") pod \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") "
Oct 30 22:46:52.323090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.322914    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\" (UniqueName: \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\") pod \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") "
Oct 30 22:46:52.323090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.322961    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\") pod \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\" (UID: \"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184\") "
Oct 30 22:46:52.323598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.323547    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca" pod "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" (UID: "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:52.325471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.325439    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd" pod "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" (UID: "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184"). InnerVolumeSpecName "kube-api-access-tb2jd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:52.325559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.325472    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca" pod "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" (UID: "daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:52.424283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.424247    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:52.424283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.424280    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb2jd\" (UniqueName: \"kubernetes.io/projected/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-kube-api-access-tb2jd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:52.424283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:52.424290    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:53.054186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.054154    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp" event={"ID":"daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184","Type":"ContainerDied","Data":"53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25"}
Oct 30 22:46:53.054372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.054194    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"
Oct 30 22:46:53.054372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.054205    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b7c84cb0ed2c69d181ab440735b40863a7f6d69080d1ba32e5a489cf58cb25"
Oct 30 22:46:53.166185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.166149    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"]
Oct 30 22:46:53.167309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.167282    4882 cpu_manager.go:414] "containerMap: removing container" podUID="daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" containerName="revoke-keys"
Oct 30 22:46:53.167309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.167309    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" containerName="revoke-keys"
Oct 30 22:46:53.167527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.167513    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" containerName="revoke-keys"
Oct 30 22:46:53.172150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.172126    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.191756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.191725    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"]
Oct 30 22:46:53.225258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.225134    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:46:53.226034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.226019    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:46:53.336636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.336598    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f58s4\" (UniqueName: \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.336825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.336678    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.336825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.336745    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.438413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.438374    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.438579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.438442    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.438579 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.438565    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f58s4\" (UniqueName: \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.439106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.439077    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.440948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.440921    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.501027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.500949    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f58s4\" (UniqueName: \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\") pod \"maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:53.790443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:53.790351    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:54.166049 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:46:54.166018    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0a3ed52_323d_4f32_840d_5ea6f9accd6a.slice/crio-0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf WatchSource:0}: Error finding container 0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf: Status 404 returned error can't find the container with id 0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf
Oct 30 22:46:54.168965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:54.168935    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"]
Oct 30 22:46:55.005822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:55.005779    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:46:55.069882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:55.069844    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6" event={"ID":"e0a3ed52-323d-4f32-840d-5ea6f9accd6a","Type":"ContainerStarted","Data":"3f54677d1c537a31f0acca584139cd8a72af58a12954c517d014c286706ddf8c"}
Oct 30 22:46:55.070073 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:55.069907    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6" event={"ID":"e0a3ed52-323d-4f32-840d-5ea6f9accd6a","Type":"ContainerStarted","Data":"0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf"}
Oct 30 22:46:55.194003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:55.193950    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6" podStartSLOduration=2.193935093 podStartE2EDuration="2.193935093s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:46:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:46:55.191150548 +0000 UTC m=+4138.364474975" watchObservedRunningTime="2026-10-30 22:46:55.193935093 +0000 UTC m=+4138.367259519"
Oct 30 22:46:56.084412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:56.084347    4882 generic.go:356] "Generic (PLEG): container finished" podID="e0a3ed52-323d-4f32-840d-5ea6f9accd6a" containerID="3f54677d1c537a31f0acca584139cd8a72af58a12954c517d014c286706ddf8c" exitCode=0
Oct 30 22:46:56.085040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:56.084527    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6" event={"ID":"e0a3ed52-323d-4f32-840d-5ea6f9accd6a","Type":"ContainerDied","Data":"3f54677d1c537a31f0acca584139cd8a72af58a12954c517d014c286706ddf8c"}
Oct 30 22:46:57.033697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.033648    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:57.033939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.033648    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:46:57.248450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.248421    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:57.389386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.389343    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\" (UniqueName: \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\") pod \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") "
Oct 30 22:46:57.389639 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.389405    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\") pod \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") "
Oct 30 22:46:57.389747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.389632    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\") pod \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\" (UID: \"e0a3ed52-323d-4f32-840d-5ea6f9accd6a\") "
Oct 30 22:46:57.389842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.389815    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca" pod "e0a3ed52-323d-4f32-840d-5ea6f9accd6a" (UID: "e0a3ed52-323d-4f32-840d-5ea6f9accd6a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:46:57.390163 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.390143    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:57.391790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.391760    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4" pod "e0a3ed52-323d-4f32-840d-5ea6f9accd6a" (UID: "e0a3ed52-323d-4f32-840d-5ea6f9accd6a"). InnerVolumeSpecName "kube-api-access-f58s4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:46:57.392134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.392116    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca" pod "e0a3ed52-323d-4f32-840d-5ea6f9accd6a" (UID: "e0a3ed52-323d-4f32-840d-5ea6f9accd6a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:46:57.491452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.491410    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:57.491452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:57.491445    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f58s4\" (UniqueName: \"kubernetes.io/projected/e0a3ed52-323d-4f32-840d-5ea6f9accd6a-kube-api-access-f58s4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:46:58.100407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:58.100366    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6" event={"ID":"e0a3ed52-323d-4f32-840d-5ea6f9accd6a","Type":"ContainerDied","Data":"0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf"}
Oct 30 22:46:58.100407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:58.100383    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"
Oct 30 22:46:58.100681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:46:58.100422    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0a2dc4b44f949e5ad933a182ff9a7a324b591e816e4f21de82135cab051b0baf"
Oct 30 22:47:00.229780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.229745    4882 scope.go:122] "RemoveContainer" containerID="4951b04eafed3791d01ae8527f00d4625d56402bbeb1190ae59eca668f832f31"
Oct 30 22:47:00.240189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.240166    4882 scope.go:122] "RemoveContainer" containerID="ed6b72ae849c7a5c1b431cbbb53ae01c5dd63aaa41e502c857fa8bacf8d4ffae"
Oct 30 22:47:00.249633 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.249612    4882 scope.go:122] "RemoveContainer" containerID="6fcf4c9254c06515a8efc3ea90156ae4e7963c213511f98061e4da9728c1783f"
Oct 30 22:47:00.258705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.258679    4882 scope.go:122] "RemoveContainer" containerID="636b40eaaded917aafd3aafa18dcb7a7fe5f356dd7f314be64e7ae84a0ae8f0a"
Oct 30 22:47:00.537448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.537355    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:47:00.646792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:00.646756    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:47:04.639029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:04.638991    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:47:04.748723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:04.748686    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:47:05.538046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.537998    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" podUID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" containerName="payload-processing" containerID="cri-o://aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905" gracePeriod=25
Oct 30 22:47:05.647968 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.647925    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" podUID="d25af3c2-a039-44d4-a197-1a726eefb161" containerName="payload-pre-processing" containerID="cri-o://a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869" gracePeriod=25
Oct 30 22:47:05.815375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.815342    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:47:05.881295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.881264    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\" (UniqueName: \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\") pod \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") "
Oct 30 22:47:05.881495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.881327    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\") pod \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\" (UID: \"480b7a25-cb3c-44b9-a04f-861d1f7cd91a\") "
Oct 30 22:47:05.882687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.882099    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume" pod "480b7a25-cb3c-44b9-a04f-861d1f7cd91a" (UID: "480b7a25-cb3c-44b9-a04f-861d1f7cd91a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:05.899271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.899233    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd" pod "480b7a25-cb3c-44b9-a04f-861d1f7cd91a" (UID: "480b7a25-cb3c-44b9-a04f-861d1f7cd91a"). InnerVolumeSpecName "kube-api-access-pjdtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:05.932439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.932408    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:47:05.983046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.983008    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\") pod \"d25af3c2-a039-44d4-a197-1a726eefb161\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") "
Oct 30 22:47:05.983260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.983167    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\" (UniqueName: \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\") pod \"d25af3c2-a039-44d4-a197-1a726eefb161\" (UID: \"d25af3c2-a039-44d4-a197-1a726eefb161\") "
Oct 30 22:47:05.983430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.983416    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjdtd\" (UniqueName: \"kubernetes.io/projected/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-kube-api-access-pjdtd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:05.983507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.983429    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/480b7a25-cb3c-44b9-a04f-861d1f7cd91a-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:05.983507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.983448    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume" pod "d25af3c2-a039-44d4-a197-1a726eefb161" (UID: "d25af3c2-a039-44d4-a197-1a726eefb161"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:05.985327 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:05.985302    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq" pod "d25af3c2-a039-44d4-a197-1a726eefb161" (UID: "d25af3c2-a039-44d4-a197-1a726eefb161"). InnerVolumeSpecName "kube-api-access-zmpvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:06.084316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.084281    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmpvq\" (UniqueName: \"kubernetes.io/projected/d25af3c2-a039-44d4-a197-1a726eefb161-kube-api-access-zmpvq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:06.084316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.084312    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d25af3c2-a039-44d4-a197-1a726eefb161-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:06.154776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.154730    4882 generic.go:356] "Generic (PLEG): container finished" podID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" containerID="aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905" exitCode=0
Oct 30 22:47:06.154992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.154799    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"
Oct 30 22:47:06.154992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.154806    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" event={"ID":"480b7a25-cb3c-44b9-a04f-861d1f7cd91a","Type":"ContainerDied","Data":"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"}
Oct 30 22:47:06.154992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.154870    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw" event={"ID":"480b7a25-cb3c-44b9-a04f-861d1f7cd91a","Type":"ContainerDied","Data":"201d09a948b715fa2684d91127ced4c358cc071772dfd63172eae8d1189aa996"}
Oct 30 22:47:06.154992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.154900    4882 scope.go:122] "RemoveContainer" containerID="aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"
Oct 30 22:47:06.156578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.156540    4882 generic.go:356] "Generic (PLEG): container finished" podID="d25af3c2-a039-44d4-a197-1a726eefb161" containerID="a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869" exitCode=0
Oct 30 22:47:06.156733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.156625    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"
Oct 30 22:47:06.156733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.156625    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" event={"ID":"d25af3c2-a039-44d4-a197-1a726eefb161","Type":"ContainerDied","Data":"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"}
Oct 30 22:47:06.156859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.156751    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2" event={"ID":"d25af3c2-a039-44d4-a197-1a726eefb161","Type":"ContainerDied","Data":"ef14e718c24a7b28c7cd3435d144f7713f9eae1f4e8b1493856115defb3f9f08"}
Oct 30 22:47:06.167656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.167628    4882 scope.go:122] "RemoveContainer" containerID="aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"
Oct 30 22:47:06.168066 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:06.167990    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905\": container with ID starting with aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905 not found: ID does not exist" containerID="aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"
Oct 30 22:47:06.168066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.168035    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905"} err="failed to get container status \"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905\": rpc error: code = NotFound desc = could not find container \"aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905\": container with ID starting with aa0f1eda521d99e67d426f9178a64036f23b6cf2052c3da4303a0b73673ea905 not found: ID does not exist"
Oct 30 22:47:06.168066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.168060    4882 scope.go:122] "RemoveContainer" containerID="a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"
Oct 30 22:47:06.179306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.179285    4882 scope.go:122] "RemoveContainer" containerID="a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"
Oct 30 22:47:06.179667 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:06.179643    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869\": container with ID starting with a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869 not found: ID does not exist" containerID="a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"
Oct 30 22:47:06.179759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.179684    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869"} err="failed to get container status \"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869\": rpc error: code = NotFound desc = could not find container \"a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869\": container with ID starting with a70597ea5591ecd5dc10c0ad2d89c4f5ef972a41755319ced5d362a215451869 not found: ID does not exist"
Oct 30 22:47:06.322362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.322319    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:47:06.326289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.326260    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-5a5534-84b4d9b9d7-2rmz2"]
Oct 30 22:47:06.438347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.438273    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:47:06.441982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.441955    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-5a5534-7ffd856679-24hkw"]
Oct 30 22:47:06.943930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.943898    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" path="/var/lib/kubelet/pods/480b7a25-cb3c-44b9-a04f-861d1f7cd91a/volumes"
Oct 30 22:47:06.944328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:06.944319    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d25af3c2-a039-44d4-a197-1a726eefb161" path="/var/lib/kubelet/pods/d25af3c2-a039-44d4-a197-1a726eefb161/volumes"
Oct 30 22:47:07.033539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:07.033502    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:07.033751 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:07.033502    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:09.099221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.099193    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:47:09.119245 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.119213    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\" (UniqueName: \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\") pod \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") "
Oct 30 22:47:09.119428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.119257    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\") pod \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\" (UID: \"e017891f-d2b9-41ad-9a23-ca0b6c26dbed\") "
Oct 30 22:47:09.119671 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.119647    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume" pod "e017891f-d2b9-41ad-9a23-ca0b6c26dbed" (UID: "e017891f-d2b9-41ad-9a23-ca0b6c26dbed"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:09.122211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.122185    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv" pod "e017891f-d2b9-41ad-9a23-ca0b6c26dbed" (UID: "e017891f-d2b9-41ad-9a23-ca0b6c26dbed"). InnerVolumeSpecName "kube-api-access-5kbcv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:09.176668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.176642    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:47:09.179506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.179478    4882 generic.go:356] "Generic (PLEG): container finished" podID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerID="625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e" exitCode=0
Oct 30 22:47:09.179659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.179520    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"
Oct 30 22:47:09.179659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.179558    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" event={"ID":"e017891f-d2b9-41ad-9a23-ca0b6c26dbed","Type":"ContainerDied","Data":"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"}
Oct 30 22:47:09.179659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.179636    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l" event={"ID":"e017891f-d2b9-41ad-9a23-ca0b6c26dbed","Type":"ContainerDied","Data":"ce5402c716ca691dfa49037ec5a2bf0597e15f808b8bdcce7e8f1687fa261a17"}
Oct 30 22:47:09.179824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.179662    4882 scope.go:122] "RemoveContainer" containerID="625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"
Oct 30 22:47:09.181164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.181130    4882 generic.go:356] "Generic (PLEG): container finished" podID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerID="4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1" exitCode=0
Oct 30 22:47:09.181277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.181174    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"
Oct 30 22:47:09.181277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.181207    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" event={"ID":"1202b53e-b96a-4525-a84f-260a0b31c8d0","Type":"ContainerDied","Data":"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"}
Oct 30 22:47:09.181277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.181252    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb" event={"ID":"1202b53e-b96a-4525-a84f-260a0b31c8d0","Type":"ContainerDied","Data":"c64dbea6f9d92d9df799a835d2131072e1ed51b51c51948b1048c10ce00df868"}
Oct 30 22:47:09.189998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.189981    4882 scope.go:122] "RemoveContainer" containerID="625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"
Oct 30 22:47:09.190288 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:09.190270    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e\": container with ID starting with 625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e not found: ID does not exist" containerID="625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"
Oct 30 22:47:09.190342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.190295    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e"} err="failed to get container status \"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e\": rpc error: code = NotFound desc = could not find container \"625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e\": container with ID starting with 625595114b9409156dc333ca86a997d51d27a61ca59a38d68cb2c916fe8b9f6e not found: ID does not exist"
Oct 30 22:47:09.190342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.190315    4882 scope.go:122] "RemoveContainer" containerID="4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"
Oct 30 22:47:09.199967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.199946    4882 scope.go:122] "RemoveContainer" containerID="4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"
Oct 30 22:47:09.200233 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:09.200213    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1\": container with ID starting with 4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1 not found: ID does not exist" containerID="4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"
Oct 30 22:47:09.200287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.200244    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1"} err="failed to get container status \"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1\": rpc error: code = NotFound desc = could not find container \"4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1\": container with ID starting with 4e4311edff9dbedf22ec583dc699e4b7cabe0a5d41267afef54c7619e54d74b1 not found: ID does not exist"
Oct 30 22:47:09.220191 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220161    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\") pod \"1202b53e-b96a-4525-a84f-260a0b31c8d0\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") "
Oct 30 22:47:09.220356 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220298    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\" (UniqueName: \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\") pod \"1202b53e-b96a-4525-a84f-260a0b31c8d0\" (UID: \"1202b53e-b96a-4525-a84f-260a0b31c8d0\") "
Oct 30 22:47:09.220599 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220560    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume" pod "1202b53e-b96a-4525-a84f-260a0b31c8d0" (UID: "1202b53e-b96a-4525-a84f-260a0b31c8d0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:09.220861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220841    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5kbcv\" (UniqueName: \"kubernetes.io/projected/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-kube-api-access-5kbcv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:09.220951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220866    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1202b53e-b96a-4525-a84f-260a0b31c8d0-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:09.220951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.220882    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e017891f-d2b9-41ad-9a23-ca0b6c26dbed-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:09.222423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.222398    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv" pod "1202b53e-b96a-4525-a84f-260a0b31c8d0" (UID: "1202b53e-b96a-4525-a84f-260a0b31c8d0"). InnerVolumeSpecName "kube-api-access-nrtrv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:09.321697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.321573    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrtrv\" (UniqueName: \"kubernetes.io/projected/1202b53e-b96a-4525-a84f-260a0b31c8d0-kube-api-access-nrtrv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:09.448616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.448555    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:47:09.454622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.454579    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-ef00a4-79d54c497f-f776l"]
Oct 30 22:47:09.605021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.604987    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:47:09.608843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.608813    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-ef00a4-7655877c99-zprzb"]
Oct 30 22:47:09.639747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.639705    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" podUID="24161188-f356-4958-afdd-c972f05fb851" containerName="payload-processing" containerID="cri-o://f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc" gracePeriod=25
Oct 30 22:47:09.749352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.749308    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" podUID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" containerName="payload-pre-processing" containerID="cri-o://ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6" gracePeriod=25
Oct 30 22:47:09.911395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:09.911361    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:47:10.029949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.029857    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\") pod \"24161188-f356-4958-afdd-c972f05fb851\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") "
Oct 30 22:47:10.030193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.030166    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\" (UniqueName: \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\") pod \"24161188-f356-4958-afdd-c972f05fb851\" (UID: \"24161188-f356-4958-afdd-c972f05fb851\") "
Oct 30 22:47:10.032360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.032325    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume" pod "24161188-f356-4958-afdd-c972f05fb851" (UID: "24161188-f356-4958-afdd-c972f05fb851"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:10.033859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.033831    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj" pod "24161188-f356-4958-afdd-c972f05fb851" (UID: "24161188-f356-4958-afdd-c972f05fb851"). InnerVolumeSpecName "kube-api-access-kcknj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:10.048580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.048552    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:47:10.131950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.131853    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\" (UniqueName: \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\") pod \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") "
Oct 30 22:47:10.132412 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.132021    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\") pod \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\" (UID: \"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf\") "
Oct 30 22:47:10.132474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.132400    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume" pod "fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" (UID: "fe4786e8-2760-4354-a1bf-8e5fc9cb8abf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:10.132545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.132511    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:10.132545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.132535    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kcknj\" (UniqueName: \"kubernetes.io/projected/24161188-f356-4958-afdd-c972f05fb851-kube-api-access-kcknj\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:10.132659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.132551    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/24161188-f356-4958-afdd-c972f05fb851-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:10.134022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.134003    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h" pod "fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" (UID: "fe4786e8-2760-4354-a1bf-8e5fc9cb8abf"). InnerVolumeSpecName "kube-api-access-2ch5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:10.190425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.190383    4882 generic.go:356] "Generic (PLEG): container finished" podID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" containerID="ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6" exitCode=0
Oct 30 22:47:10.190612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.190431    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"
Oct 30 22:47:10.190612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.190469    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" event={"ID":"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf","Type":"ContainerDied","Data":"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"}
Oct 30 22:47:10.190612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.190539    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd" event={"ID":"fe4786e8-2760-4354-a1bf-8e5fc9cb8abf","Type":"ContainerDied","Data":"0b897f6e883ab70e8406752194ce5795c409cec93c697307560086bf051edd09"}
Oct 30 22:47:10.190612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.190573    4882 scope.go:122] "RemoveContainer" containerID="ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"
Oct 30 22:47:10.193972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.193939    4882 generic.go:356] "Generic (PLEG): container finished" podID="24161188-f356-4958-afdd-c972f05fb851" containerID="f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc" exitCode=0
Oct 30 22:47:10.194102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.193996    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"
Oct 30 22:47:10.194102 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.194029    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" event={"ID":"24161188-f356-4958-afdd-c972f05fb851","Type":"ContainerDied","Data":"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"}
Oct 30 22:47:10.194237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.194098    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7" event={"ID":"24161188-f356-4958-afdd-c972f05fb851","Type":"ContainerDied","Data":"254e93f36e1a881a4b5c74dc36064337cc5ebd264d0745963c9e65db344add39"}
Oct 30 22:47:10.209689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.209661    4882 scope.go:122] "RemoveContainer" containerID="ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"
Oct 30 22:47:10.210001 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:10.209980    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6\": container with ID starting with ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6 not found: ID does not exist" containerID="ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"
Oct 30 22:47:10.210070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.210011    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6"} err="failed to get container status \"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6\": rpc error: code = NotFound desc = could not find container \"ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6\": container with ID starting with ba26d84cb8867e0a3d9a2e8796052b5d3f4edc364905ba24bfa509a1629afdf6 not found: ID does not exist"
Oct 30 22:47:10.210070 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.210027    4882 scope.go:122] "RemoveContainer" containerID="f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"
Oct 30 22:47:10.219790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.219769    4882 scope.go:122] "RemoveContainer" containerID="f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"
Oct 30 22:47:10.220071 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:10.220052    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc\": container with ID starting with f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc not found: ID does not exist" containerID="f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"
Oct 30 22:47:10.220120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.220081    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc"} err="failed to get container status \"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc\": rpc error: code = NotFound desc = could not find container \"f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc\": container with ID starting with f0d095c71c724114702afee17a9eba60aeb9acedc03fcdf03cb97ab51f57a9fc not found: ID does not exist"
Oct 30 22:47:10.233886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.233848    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ch5h\" (UniqueName: \"kubernetes.io/projected/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf-kube-api-access-2ch5h\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:10.358320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.358282    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:47:10.361813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.361777    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-8c852b-6587d75fd7-c49hd"]
Oct 30 22:47:10.472624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.472523    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:47:10.476047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.476016    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-8c852b-75b86db447-x5vk7"]
Oct 30 22:47:10.943838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.943806    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" path="/var/lib/kubelet/pods/1202b53e-b96a-4525-a84f-260a0b31c8d0/volumes"
Oct 30 22:47:10.944287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.944271    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24161188-f356-4958-afdd-c972f05fb851" path="/var/lib/kubelet/pods/24161188-f356-4958-afdd-c972f05fb851/volumes"
Oct 30 22:47:10.944732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.944717    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" path="/var/lib/kubelet/pods/e017891f-d2b9-41ad-9a23-ca0b6c26dbed/volumes"
Oct 30 22:47:10.945133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:10.945120    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" path="/var/lib/kubelet/pods/fe4786e8-2760-4354-a1bf-8e5fc9cb8abf/volumes"
Oct 30 22:47:14.522524 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:14.522488    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"]
Oct 30 22:47:14.572702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:14.572663    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-5a5534-x76mp"]
Oct 30 22:47:14.944018 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:14.943983    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184" path="/var/lib/kubelet/pods/daf67d68-ab4e-4d8f-bd07-3e8c6fb4f184/volumes"
Oct 30 22:47:17.790843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.790807    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"]
Oct 30 22:47:17.791435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791420    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing"
Oct 30 22:47:17.791481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791436    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing"
Oct 30 22:47:17.791481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791459    4882 cpu_manager.go:414] "containerMap: removing container" podUID="24161188-f356-4958-afdd-c972f05fb851" containerName="payload-processing"
Oct 30 22:47:17.791481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791464    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24161188-f356-4958-afdd-c972f05fb851" containerName="payload-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791488    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d25af3c2-a039-44d4-a197-1a726eefb161" containerName="payload-pre-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791494    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d25af3c2-a039-44d4-a197-1a726eefb161" containerName="payload-pre-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791502    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e0a3ed52-323d-4f32-840d-5ea6f9accd6a" containerName="revoke-keys"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791507    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0a3ed52-323d-4f32-840d-5ea6f9accd6a" containerName="revoke-keys"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791519    4882 cpu_manager.go:414] "containerMap: removing container" podUID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" containerName="payload-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791525    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" containerName="payload-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791532    4882 cpu_manager.go:414] "containerMap: removing container" podUID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" containerName="payload-pre-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791536    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" containerName="payload-pre-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791546    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing"
Oct 30 22:47:17.791619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791551    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791660    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="24161188-f356-4958-afdd-c972f05fb851" containerName="payload-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791674    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d25af3c2-a039-44d4-a197-1a726eefb161" containerName="payload-pre-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791682    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0a3ed52-323d-4f32-840d-5ea6f9accd6a" containerName="revoke-keys"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791688    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="480b7a25-cb3c-44b9-a04f-861d1f7cd91a" containerName="payload-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791694    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe4786e8-2760-4354-a1bf-8e5fc9cb8abf" containerName="payload-pre-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791700    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1202b53e-b96a-4525-a84f-260a0b31c8d0" containerName="payload-pre-processing"
Oct 30 22:47:17.791947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.791707    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e017891f-d2b9-41ad-9a23-ca0b6c26dbed" containerName="payload-processing"
Oct 30 22:47:17.796509 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.796482    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:17.849300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.849262    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"]
Oct 30 22:47:17.895275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.895235    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:17.895452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.895357    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:47:17.919904 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.919865    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:17.920133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.920033    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:17.920133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:17.920119    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwr8m\" (UniqueName: \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.021114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.021073    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.021319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.021197    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwr8m\" (UniqueName: \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.021520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.021499    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.021936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.021905    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.024083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.024056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.083994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.083960    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwr8m\" (UniqueName: \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\") pod \"maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.110701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.110667    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:18.485128 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:47:18.485093    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d46bc4_382e_4728_9e56_c13cab6415b6.slice/crio-7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb WatchSource:0}: Error finding container 7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb: Status 404 returned error can't find the container with id 7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb
Oct 30 22:47:18.487266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:18.487240    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"]
Oct 30 22:47:19.272013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:19.271976    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4" event={"ID":"43d46bc4-382e-4728-9e56-c13cab6415b6","Type":"ContainerStarted","Data":"4fc253a7632fd77cc5103f119901a7fd35279ca18c1432c66c05e94b9fceebed"}
Oct 30 22:47:19.272420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:19.272034    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4" event={"ID":"43d46bc4-382e-4728-9e56-c13cab6415b6","Type":"ContainerStarted","Data":"7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb"}
Oct 30 22:47:19.392878 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:19.392816    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4" podStartSLOduration=2.392800283 podStartE2EDuration="2.392800283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:19.389154023 +0000 UTC m=+4162.562478450" watchObservedRunningTime="2026-10-30 22:47:19.392800283 +0000 UTC m=+4162.566124710"
Oct 30 22:47:19.759032 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:19.758996    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d46bc4_382e_4728_9e56_c13cab6415b6.slice/crio-conmon-4fc253a7632fd77cc5103f119901a7fd35279ca18c1432c66c05e94b9fceebed.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:47:20.279999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:20.279961    4882 generic.go:356] "Generic (PLEG): container finished" podID="43d46bc4-382e-4728-9e56-c13cab6415b6" containerID="4fc253a7632fd77cc5103f119901a7fd35279ca18c1432c66c05e94b9fceebed" exitCode=0
Oct 30 22:47:20.280427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:20.280051    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4" event={"ID":"43d46bc4-382e-4728-9e56-c13cab6415b6","Type":"ContainerDied","Data":"4fc253a7632fd77cc5103f119901a7fd35279ca18c1432c66c05e94b9fceebed"}
Oct 30 22:47:20.807827 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:20.807773    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:47:20.899858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:20.899820    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"]
Oct 30 22:47:20.903240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:20.903214    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w1-ef00a4-gnn74"]
Oct 30 22:47:20.943945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:20.943910    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6835e137-f50d-42bd-82c9-2e037a59e63f" path="/var/lib/kubelet/pods/6835e137-f50d-42bd-82c9-2e037a59e63f/volumes"
Oct 30 22:47:21.445007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.444978    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:21.561110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.561019    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\") pod \"43d46bc4-382e-4728-9e56-c13cab6415b6\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") "
Oct 30 22:47:21.561285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.561127    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\") pod \"43d46bc4-382e-4728-9e56-c13cab6415b6\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") "
Oct 30 22:47:21.561285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.561178    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\" (UniqueName: \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\") pod \"43d46bc4-382e-4728-9e56-c13cab6415b6\" (UID: \"43d46bc4-382e-4728-9e56-c13cab6415b6\") "
Oct 30 22:47:21.561520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.561484    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca" pod "43d46bc4-382e-4728-9e56-c13cab6415b6" (UID: "43d46bc4-382e-4728-9e56-c13cab6415b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:21.561763 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.561735    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:21.563308 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.563281    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca" pod "43d46bc4-382e-4728-9e56-c13cab6415b6" (UID: "43d46bc4-382e-4728-9e56-c13cab6415b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:21.563542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.563514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m" pod "43d46bc4-382e-4728-9e56-c13cab6415b6" (UID: "43d46bc4-382e-4728-9e56-c13cab6415b6"). InnerVolumeSpecName "kube-api-access-wwr8m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:21.663246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.663202    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/43d46bc4-382e-4728-9e56-c13cab6415b6-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:21.663246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:21.663238    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwr8m\" (UniqueName: \"kubernetes.io/projected/43d46bc4-382e-4728-9e56-c13cab6415b6-kube-api-access-wwr8m\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:22.294421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:22.294386    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"
Oct 30 22:47:22.294658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:22.294424    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4" event={"ID":"43d46bc4-382e-4728-9e56-c13cab6415b6","Type":"ContainerDied","Data":"7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb"}
Oct 30 22:47:22.294658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:22.294487    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad4d95d9c4cfc9665b596dc268441c03909a3fcbbd9f3723a18694da48811bb"
Oct 30 22:47:26.064709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:26.064672    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"]
Oct 30 22:47:26.115259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:26.115221    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c44ede00-7qzkw"]
Oct 30 22:47:26.944629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:26.944576    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="951e3eec-6db5-49eb-a7ad-8df628ddbb25" path="/var/lib/kubelet/pods/951e3eec-6db5-49eb-a7ad-8df628ddbb25/volumes"
Oct 30 22:47:29.064932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.064898    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"]
Oct 30 22:47:29.065532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.065517    4882 cpu_manager.go:414] "containerMap: removing container" podUID="43d46bc4-382e-4728-9e56-c13cab6415b6" containerName="revoke-keys"
Oct 30 22:47:29.065601 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.065533    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d46bc4-382e-4728-9e56-c13cab6415b6" containerName="revoke-keys"
Oct 30 22:47:29.065690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.065679    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d46bc4-382e-4728-9e56-c13cab6415b6" containerName="revoke-keys"
Oct 30 22:47:29.070486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.070467    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.123377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.123338    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"]
Oct 30 22:47:29.169554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.169517    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:29.169748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.169562    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:47:29.249615 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.249558    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.249833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.249700    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.249833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.249826    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9flfr\" (UniqueName: \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.350755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.350723    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.350982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.350764    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.350982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.350844    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9flfr\" (UniqueName: \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.351547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.351521    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.353303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.353267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.412281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.412243    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9flfr\" (UniqueName: \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\") pod \"maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:29.684577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:29.684493    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:30.057074 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:47:30.057037    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12caa3cc_3532_4a6d_9d45_e1dd9d94d13d.slice/crio-eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31 WatchSource:0}: Error finding container eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31: Status 404 returned error can't find the container with id eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31
Oct 30 22:47:30.061152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.060999    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"]
Oct 30 22:47:30.357187 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.357152    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl" event={"ID":"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d","Type":"ContainerStarted","Data":"bdd083731890629e386a3dd0f65d146622edb2e735429fb5b9d3dec5d6f4c836"}
Oct 30 22:47:30.357562 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.357215    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl" event={"ID":"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d","Type":"ContainerStarted","Data":"eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31"}
Oct 30 22:47:30.475715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.475662    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl" podStartSLOduration=1.475647027 podStartE2EDuration="1.475647027s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:30.47512801 +0000 UTC m=+4173.648452439" watchObservedRunningTime="2026-10-30 22:47:30.475647027 +0000 UTC m=+4173.648971453"
Oct 30 22:47:30.653936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.653850    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:47:30.772903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:30.772870    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:47:31.366571 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:31.366538    4882 generic.go:356] "Generic (PLEG): container finished" podID="12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" containerID="bdd083731890629e386a3dd0f65d146622edb2e735429fb5b9d3dec5d6f4c836" exitCode=0
Oct 30 22:47:31.367021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:31.366644    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl" event={"ID":"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d","Type":"ContainerDied","Data":"bdd083731890629e386a3dd0f65d146622edb2e735429fb5b9d3dec5d6f4c836"}
Oct 30 22:47:32.525140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.525111    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:32.594146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.594112    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\") pod \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") "
Oct 30 22:47:32.594370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.594310    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\") pod \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") "
Oct 30 22:47:32.594370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.594345    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\" (UniqueName: \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\") pod \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\" (UID: \"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d\") "
Oct 30 22:47:32.594744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.594714    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca" pod "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" (UID: "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:32.596265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.596241    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca" pod "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" (UID: "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:32.596484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.596464    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr" pod "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" (UID: "12caa3cc-3532-4a6d-9d45-e1dd9d94d13d"). InnerVolumeSpecName "kube-api-access-9flfr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:32.695413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.695331    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.695413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.695360    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9flfr\" (UniqueName: \"kubernetes.io/projected/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-kube-api-access-9flfr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:32.695413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:32.695370    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:33.381319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:33.381281    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"
Oct 30 22:47:33.381319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:33.381313    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl" event={"ID":"12caa3cc-3532-4a6d-9d45-e1dd9d94d13d","Type":"ContainerDied","Data":"eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31"}
Oct 30 22:47:33.381622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:33.381361    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eec7d10af6cdf75b55d891ce1f00472bf96f0f1f8290a087de7ac0ff8f4d0b31"
Oct 30 22:47:35.654242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:35.654177    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" podUID="42733af8-2041-4ee3-b31a-31674d10ee76" containerName="payload-processing" containerID="cri-o://800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd" gracePeriod=25
Oct 30 22:47:35.773909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:35.773871    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" podUID="6510e6c7-e649-4743-a8dd-8ea99a26356d" containerName="payload-pre-processing" containerID="cri-o://cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d" gracePeriod=25
Oct 30 22:47:35.934201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:35.934168    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:47:36.031250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.031099    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\" (UniqueName: \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\") pod \"42733af8-2041-4ee3-b31a-31674d10ee76\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") "
Oct 30 22:47:36.031432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.031330    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\") pod \"42733af8-2041-4ee3-b31a-31674d10ee76\" (UID: \"42733af8-2041-4ee3-b31a-31674d10ee76\") "
Oct 30 22:47:36.031754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.031723    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume" pod "42733af8-2041-4ee3-b31a-31674d10ee76" (UID: "42733af8-2041-4ee3-b31a-31674d10ee76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:36.032148 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.032112    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42733af8-2041-4ee3-b31a-31674d10ee76-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:36.033643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.033620    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms" pod "42733af8-2041-4ee3-b31a-31674d10ee76" (UID: "42733af8-2041-4ee3-b31a-31674d10ee76"). InnerVolumeSpecName "kube-api-access-b9tms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:36.044415 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.044392    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:47:36.052498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.052472    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"]
Oct 30 22:47:36.106977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.106942    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0ac779d3-hbs6q"]
Oct 30 22:47:36.132687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.132655    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\") pod \"6510e6c7-e649-4743-a8dd-8ea99a26356d\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") "
Oct 30 22:47:36.132870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.132790    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\" (UniqueName: \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\") pod \"6510e6c7-e649-4743-a8dd-8ea99a26356d\" (UID: \"6510e6c7-e649-4743-a8dd-8ea99a26356d\") "
Oct 30 22:47:36.133099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.133074    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9tms\" (UniqueName: \"kubernetes.io/projected/42733af8-2041-4ee3-b31a-31674d10ee76-kube-api-access-b9tms\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:36.133099 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.133081    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume" pod "6510e6c7-e649-4743-a8dd-8ea99a26356d" (UID: "6510e6c7-e649-4743-a8dd-8ea99a26356d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:36.134936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.134917    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf" pod "6510e6c7-e649-4743-a8dd-8ea99a26356d" (UID: "6510e6c7-e649-4743-a8dd-8ea99a26356d"). InnerVolumeSpecName "kube-api-access-w4wcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:36.234077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.233996    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wcf\" (UniqueName: \"kubernetes.io/projected/6510e6c7-e649-4743-a8dd-8ea99a26356d-kube-api-access-w4wcf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:36.234077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.234027    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6510e6c7-e649-4743-a8dd-8ea99a26356d-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:36.405459 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.405418    4882 generic.go:356] "Generic (PLEG): container finished" podID="42733af8-2041-4ee3-b31a-31674d10ee76" containerID="800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd" exitCode=0
Oct 30 22:47:36.405687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.405479    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"
Oct 30 22:47:36.405687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.405483    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" event={"ID":"42733af8-2041-4ee3-b31a-31674d10ee76","Type":"ContainerDied","Data":"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"}
Oct 30 22:47:36.405687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.405534    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db" event={"ID":"42733af8-2041-4ee3-b31a-31674d10ee76","Type":"ContainerDied","Data":"624c4c4ad124c3599695808e64d409f60e683835701a490cbd9d512721ab2cbf"}
Oct 30 22:47:36.405687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.405556    4882 scope.go:122] "RemoveContainer" containerID="800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"
Oct 30 22:47:36.407101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.407075    4882 generic.go:356] "Generic (PLEG): container finished" podID="6510e6c7-e649-4743-a8dd-8ea99a26356d" containerID="cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d" exitCode=0
Oct 30 22:47:36.407228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.407119    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"
Oct 30 22:47:36.407228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.407133    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" event={"ID":"6510e6c7-e649-4743-a8dd-8ea99a26356d","Type":"ContainerDied","Data":"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"}
Oct 30 22:47:36.407337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.407234    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn" event={"ID":"6510e6c7-e649-4743-a8dd-8ea99a26356d","Type":"ContainerDied","Data":"0f8d5f1a78d8b5fe0ad7a527a08e80835089e6f1699b1f49ac1ae8c49f8aecf3"}
Oct 30 22:47:36.418936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.418791    4882 scope.go:122] "RemoveContainer" containerID="800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"
Oct 30 22:47:36.419149 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:36.419117    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd\": container with ID starting with 800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd not found: ID does not exist" containerID="800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"
Oct 30 22:47:36.419249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.419162    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd"} err="failed to get container status \"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd\": rpc error: code = NotFound desc = could not find container \"800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd\": container with ID starting with 800bf27b30e3b5fef8e1614f0d6049f29645866ee9a837ab84f00581a2585fdd not found: ID does not exist"
Oct 30 22:47:36.419249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.419187    4882 scope.go:122] "RemoveContainer" containerID="cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"
Oct 30 22:47:36.428687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.428667    4882 scope.go:122] "RemoveContainer" containerID="cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"
Oct 30 22:47:36.428963 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:36.428946    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d\": container with ID starting with cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d not found: ID does not exist" containerID="cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"
Oct 30 22:47:36.429014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.428974    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d"} err="failed to get container status \"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d\": rpc error: code = NotFound desc = could not find container \"cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d\": container with ID starting with cbd5a27117de798d6eb7d3e360e63e910b2a3c65441b8c0e7a29dba0719cac3d not found: ID does not exist"
Oct 30 22:47:36.577224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.577138    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:47:36.582272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.582245    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-6e0b99-88d6ff86f-4x7db"]
Oct 30 22:47:36.693687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.693555    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:47:36.697143 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.697118    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-6e0b99-58bd4bd48c-j78rn"]
Oct 30 22:47:36.944504 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.944470    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1655dbfa-f71b-470a-9011-5038a14007f9" path="/var/lib/kubelet/pods/1655dbfa-f71b-470a-9011-5038a14007f9/volumes"
Oct 30 22:47:36.944934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.944918    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42733af8-2041-4ee3-b31a-31674d10ee76" path="/var/lib/kubelet/pods/42733af8-2041-4ee3-b31a-31674d10ee76/volumes"
Oct 30 22:47:36.945282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.945268    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6510e6c7-e649-4743-a8dd-8ea99a26356d" path="/var/lib/kubelet/pods/6510e6c7-e649-4743-a8dd-8ea99a26356d/volumes"
Oct 30 22:47:36.985195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.985161    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"]
Oct 30 22:47:36.989112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:36.989085    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-6e0b99-c25g4"]
Oct 30 22:47:37.356383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.356345    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"]
Oct 30 22:47:37.357022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357007    4882 cpu_manager.go:414] "containerMap: removing container" podUID="42733af8-2041-4ee3-b31a-31674d10ee76" containerName="payload-processing"
Oct 30 22:47:37.357091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357024    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42733af8-2041-4ee3-b31a-31674d10ee76" containerName="payload-processing"
Oct 30 22:47:37.357091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357036    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6510e6c7-e649-4743-a8dd-8ea99a26356d" containerName="payload-pre-processing"
Oct 30 22:47:37.357091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357042    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6510e6c7-e649-4743-a8dd-8ea99a26356d" containerName="payload-pre-processing"
Oct 30 22:47:37.357091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357058    4882 cpu_manager.go:414] "containerMap: removing container" podUID="12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" containerName="revoke-keys"
Oct 30 22:47:37.357091 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357063    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" containerName="revoke-keys"
Oct 30 22:47:37.357266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357171    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="42733af8-2041-4ee3-b31a-31674d10ee76" containerName="payload-processing"
Oct 30 22:47:37.357266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357179    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6510e6c7-e649-4743-a8dd-8ea99a26356d" containerName="payload-pre-processing"
Oct 30 22:47:37.357266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.357188    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" containerName="revoke-keys"
Oct 30 22:47:37.362359 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.362339    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.415394 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.415356    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"]
Oct 30 22:47:37.447101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.447062    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.447101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.447072    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:37.447347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.447143    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.447347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.447219    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:47:37.447347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.447226    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxf8z\" (UniqueName: \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.548563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.548518    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.548822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.548573    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.548822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.548665    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxf8z\" (UniqueName: \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.549385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.549356    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.551048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.551027    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.610346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.610260    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxf8z\" (UniqueName: \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\") pod \"maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:37.677495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:37.677456    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:38.055104 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:47:38.055057    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06329c33_ce9b_4476_b427_9e31a346c0e2.slice/crio-81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3 WatchSource:0}: Error finding container 81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3: Status 404 returned error can't find the container with id 81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3
Oct 30 22:47:38.058132 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:38.058087    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"]
Oct 30 22:47:38.426288 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:38.426245    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx" event={"ID":"06329c33-ce9b-4476-b427-9e31a346c0e2","Type":"ContainerStarted","Data":"89da38e0c1f4a29f283e15b30067d6263013e2a54d7587cdb540e13bec3ef98a"}
Oct 30 22:47:38.426514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:38.426312    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx" event={"ID":"06329c33-ce9b-4476-b427-9e31a346c0e2","Type":"ContainerStarted","Data":"81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3"}
Oct 30 22:47:38.544891 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:38.544838    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx" podStartSLOduration=1.544823405 podStartE2EDuration="1.544823405s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:38.542310106 +0000 UTC m=+4181.715634534" watchObservedRunningTime="2026-10-30 22:47:38.544823405 +0000 UTC m=+4181.718147832"
Oct 30 22:47:38.945230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:38.945197    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d46bc4-382e-4728-9e56-c13cab6415b6" path="/var/lib/kubelet/pods/43d46bc4-382e-4728-9e56-c13cab6415b6/volumes"
Oct 30 22:47:39.434383 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:39.434342    4882 generic.go:356] "Generic (PLEG): container finished" podID="06329c33-ce9b-4476-b427-9e31a346c0e2" containerID="89da38e0c1f4a29f283e15b30067d6263013e2a54d7587cdb540e13bec3ef98a" exitCode=0
Oct 30 22:47:39.434854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:39.434424    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx" event={"ID":"06329c33-ce9b-4476-b427-9e31a346c0e2","Type":"ContainerDied","Data":"89da38e0c1f4a29f283e15b30067d6263013e2a54d7587cdb540e13bec3ef98a"}
Oct 30 22:47:40.585831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.585787    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:47:40.591685 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.591662    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:40.684262 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.684233    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\") pod \"06329c33-ce9b-4476-b427-9e31a346c0e2\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") "
Oct 30 22:47:40.684488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.684467    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\") pod \"06329c33-ce9b-4476-b427-9e31a346c0e2\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") "
Oct 30 22:47:40.684619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.684597    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\" (UniqueName: \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\") pod \"06329c33-ce9b-4476-b427-9e31a346c0e2\" (UID: \"06329c33-ce9b-4476-b427-9e31a346c0e2\") "
Oct 30 22:47:40.684760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.684693    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca" pod "06329c33-ce9b-4476-b427-9e31a346c0e2" (UID: "06329c33-ce9b-4476-b427-9e31a346c0e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:40.685036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.685013    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.686810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.686783    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca" pod "06329c33-ce9b-4476-b427-9e31a346c0e2" (UID: "06329c33-ce9b-4476-b427-9e31a346c0e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:40.686964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.686941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z" pod "06329c33-ce9b-4476-b427-9e31a346c0e2" (UID: "06329c33-ce9b-4476-b427-9e31a346c0e2"). InnerVolumeSpecName "kube-api-access-mxf8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:40.688098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.688071    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:47:40.786329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.786296    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06329c33-ce9b-4476-b427-9e31a346c0e2-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.786329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:40.786327    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxf8z\" (UniqueName: \"kubernetes.io/projected/06329c33-ce9b-4476-b427-9e31a346c0e2-kube-api-access-mxf8z\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:40.997632 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:40.997582    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06329c33_ce9b_4476_b427_9e31a346c0e2.slice\": RecentStats: unable to find data in memory cache]"
Oct 30 22:47:41.449845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:41.449809    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx" event={"ID":"06329c33-ce9b-4476-b427-9e31a346c0e2","Type":"ContainerDied","Data":"81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3"}
Oct 30 22:47:41.450010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:41.449861    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81541c373f71b87bbf845617a4010a709d55a3c8290263fcccb544eb85fcede3"
Oct 30 22:47:41.450010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:41.449823    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"
Oct 30 22:47:43.940300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:43.940267    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:47:43.940748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:43.940470    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" containerID="cri-o://6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4" gracePeriod=30
Oct 30 22:47:44.673179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:44.673141    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:47:45.587024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:45.586968    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing" containerID="cri-o://d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e" gracePeriod=25
Oct 30 22:47:45.688696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:45.688648    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing" containerID="cri-o://b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921" gracePeriod=25
Oct 30 22:47:46.149924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.149887    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"]
Oct 30 22:47:46.150687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.150666    4882 cpu_manager.go:414] "containerMap: removing container" podUID="06329c33-ce9b-4476-b427-9e31a346c0e2" containerName="revoke-keys"
Oct 30 22:47:46.150756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.150689    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06329c33-ce9b-4476-b427-9e31a346c0e2" containerName="revoke-keys"
Oct 30 22:47:46.150795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.150786    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="06329c33-ce9b-4476-b427-9e31a346c0e2" containerName="revoke-keys"
Oct 30 22:47:46.154449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.154432    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.209247 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.209205    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"]
Oct 30 22:47:46.249420 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.249385    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.249621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.249426    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.249621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.249560    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgts2\" (UniqueName: \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.254406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.254382    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:47:46.254478 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.254465    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:47:46.351023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.350985    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.351023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.351029    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.351287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.351093    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgts2\" (UniqueName: \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.351860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.351830    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.353449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.353428    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.413064 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.412988    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgts2\" (UniqueName: \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.468193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.468162    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:46.843728 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:47:46.843689    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d3dab08_fc99_49f3_b53b_8cf31ef9950e.slice/crio-59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566 WatchSource:0}: Error finding container 59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566: Status 404 returned error can't find the container with id 59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566
Oct 30 22:47:46.846090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:46.846064    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"]
Oct 30 22:47:47.492736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:47.492643    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf" event={"ID":"5d3dab08-fc99-49f3-b53b-8cf31ef9950e","Type":"ContainerStarted","Data":"a923d6344c5b791b57fc0d0b895dddcf0637d8227c95e3888fb0d6f9463dddb5"}
Oct 30 22:47:47.492736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:47.492702    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf" event={"ID":"5d3dab08-fc99-49f3-b53b-8cf31ef9950e","Type":"ContainerStarted","Data":"59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566"}
Oct 30 22:47:47.612752 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:47.612697    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf" podStartSLOduration=1.6126818809999999 podStartE2EDuration="1.612681881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:47.609002331 +0000 UTC m=+4190.782326758" watchObservedRunningTime="2026-10-30 22:47:47.612681881 +0000 UTC m=+4190.786006308"
Oct 30 22:47:48.501130 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.501096    4882 generic.go:356] "Generic (PLEG): container finished" podID="5d3dab08-fc99-49f3-b53b-8cf31ef9950e" containerID="a923d6344c5b791b57fc0d0b895dddcf0637d8227c95e3888fb0d6f9463dddb5" exitCode=0
Oct 30 22:47:48.501573 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.501190    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf" event={"ID":"5d3dab08-fc99-49f3-b53b-8cf31ef9950e","Type":"ContainerDied","Data":"a923d6344c5b791b57fc0d0b895dddcf0637d8227c95e3888fb0d6f9463dddb5"}
Oct 30 22:47:48.523788 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.523755    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Oct 30 22:47:48.523963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.523807    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Oct 30 22:47:48.859071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.859032    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:47:48.968136 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:48.968106    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:47:49.237621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.237581    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:47:49.283292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283255    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283353    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283391    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283432    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283534    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283600    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283645    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.283713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283682    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283726    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\") pod \"71a90030-8be8-44df-a758-a6e886b7123b\" (UID: \"71a90030-8be8-44df-a758-a6e886b7123b\") "
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283725    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.283983    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.284235    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.284257    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.284463    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.284481    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:49.285013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.284665    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:49.287199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.287165    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:47:49.288547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.288519    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:49.288775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.288655    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:47:49.288775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.288735    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m" pod "71a90030-8be8-44df-a758-a6e886b7123b" (UID: "71a90030-8be8-44df-a758-a6e886b7123b"). InnerVolumeSpecName "kube-api-access-p7n9m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:49.385741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385656    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.385741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385698    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.385741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385714    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7n9m\" (UniqueName: \"kubernetes.io/projected/71a90030-8be8-44df-a758-a6e886b7123b-kube-api-access-p7n9m\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.385741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385729    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/71a90030-8be8-44df-a758-a6e886b7123b-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.386009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385743    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/71a90030-8be8-44df-a758-a6e886b7123b-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.386009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385760    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.386009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.385773    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/71a90030-8be8-44df-a758-a6e886b7123b-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.509511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.509470    4882 generic.go:356] "Generic (PLEG): container finished" podID="71a90030-8be8-44df-a758-a6e886b7123b" containerID="6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4" exitCode=0
Oct 30 22:47:49.510048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.509519    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"
Oct 30 22:47:49.510048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.509553    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" event={"ID":"71a90030-8be8-44df-a758-a6e886b7123b","Type":"ContainerDied","Data":"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"}
Oct 30 22:47:49.510048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.509619    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6" event={"ID":"71a90030-8be8-44df-a758-a6e886b7123b","Type":"ContainerDied","Data":"9865f193450a2f0dfe364fac99a23671a0adca6798ae25f022e2fc13cc02a63b"}
Oct 30 22:47:49.510048 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.509635    4882 scope.go:122] "RemoveContainer" containerID="6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"
Oct 30 22:47:49.521487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.521459    4882 scope.go:122] "RemoveContainer" containerID="6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"
Oct 30 22:47:49.521874 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:49.521852    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4\": container with ID starting with 6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4 not found: ID does not exist" containerID="6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"
Oct 30 22:47:49.521959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.521883    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4"} err="failed to get container status \"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4\": rpc error: code = NotFound desc = could not find container \"6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4\": container with ID starting with 6c95fbdb8e0f9baf743e3def3199c5bcdf9d6f3843ffda583b1ab337d52d05b4 not found: ID does not exist"
Oct 30 22:47:49.655367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.655337    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:49.681193 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.681151    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:47:49.684728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.684700    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-6e0b99-openshift-default-849bf65785-42kd6"]
Oct 30 22:47:49.789917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.789883    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\") pod \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") "
Oct 30 22:47:49.790097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.789948    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\" (UniqueName: \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\") pod \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") "
Oct 30 22:47:49.790097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.790081    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\") pod \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\" (UID: \"5d3dab08-fc99-49f3-b53b-8cf31ef9950e\") "
Oct 30 22:47:49.790342 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.790310    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca" pod "5d3dab08-fc99-49f3-b53b-8cf31ef9950e" (UID: "5d3dab08-fc99-49f3-b53b-8cf31ef9950e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:49.790473 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.790456    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.792220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.792197    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca" pod "5d3dab08-fc99-49f3-b53b-8cf31ef9950e" (UID: "5d3dab08-fc99-49f3-b53b-8cf31ef9950e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:47:49.792313 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.792266    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2" pod "5d3dab08-fc99-49f3-b53b-8cf31ef9950e" (UID: "5d3dab08-fc99-49f3-b53b-8cf31ef9950e"). InnerVolumeSpecName "kube-api-access-mgts2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:49.891844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.891807    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:49.891844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:49.891839    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mgts2\" (UniqueName: \"kubernetes.io/projected/5d3dab08-fc99-49f3-b53b-8cf31ef9950e-kube-api-access-mgts2\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:50.307428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.307337    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:50.307428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.307351    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:47:50.518237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.518205    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"
Oct 30 22:47:50.518769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.518243    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf" event={"ID":"5d3dab08-fc99-49f3-b53b-8cf31ef9950e","Type":"ContainerDied","Data":"59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566"}
Oct 30 22:47:50.518769 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.518326    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59d3a1192ef259934b564087b6100f3b2f7e00f44d1056f10154fd6eed983566"
Oct 30 22:47:50.807754 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:50.807708    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:47:50.943946 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:50.943914    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71a90030-8be8-44df-a758-a6e886b7123b" path="/var/lib/kubelet/pods/71a90030-8be8-44df-a758-a6e886b7123b/volumes"
Oct 30 22:47:53.859550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:53.859508    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" podUID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" containerName="payload-processing" containerID="cri-o://e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200" gracePeriod=25
Oct 30 22:47:53.969395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:53.969335    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" podUID="e51de84d-55c4-43ab-bcfb-9709d87e7753" containerName="payload-pre-processing" containerID="cri-o://d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202" gracePeriod=25
Oct 30 22:47:54.127834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.127753    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:47:54.237553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.237521    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\") pod \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") "
Oct 30 22:47:54.237746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.237634    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\" (UniqueName: \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\") pod \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\" (UID: \"c0755c3c-e2a0-44e1-af29-df19118a6ef4\") "
Oct 30 22:47:54.237997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.237972    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume" pod "c0755c3c-e2a0-44e1-af29-df19118a6ef4" (UID: "c0755c3c-e2a0-44e1-af29-df19118a6ef4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:54.238089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.238069    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0755c3c-e2a0-44e1-af29-df19118a6ef4-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.240071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.240036    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l" pod "c0755c3c-e2a0-44e1-af29-df19118a6ef4" (UID: "c0755c3c-e2a0-44e1-af29-df19118a6ef4"). InnerVolumeSpecName "kube-api-access-pn84l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:54.250203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.250172    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:47:54.338678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.338644    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\") pod \"e51de84d-55c4-43ab-bcfb-9709d87e7753\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") "
Oct 30 22:47:54.338871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.338741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\" (UniqueName: \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\") pod \"e51de84d-55c4-43ab-bcfb-9709d87e7753\" (UID: \"e51de84d-55c4-43ab-bcfb-9709d87e7753\") "
Oct 30 22:47:54.339134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.339100    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume" pod "e51de84d-55c4-43ab-bcfb-9709d87e7753" (UID: "e51de84d-55c4-43ab-bcfb-9709d87e7753"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:47:54.339240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.339219    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e51de84d-55c4-43ab-bcfb-9709d87e7753-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.339320 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.339246    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pn84l\" (UniqueName: \"kubernetes.io/projected/c0755c3c-e2a0-44e1-af29-df19118a6ef4-kube-api-access-pn84l\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.341497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.341470    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb" pod "e51de84d-55c4-43ab-bcfb-9709d87e7753" (UID: "e51de84d-55c4-43ab-bcfb-9709d87e7753"). InnerVolumeSpecName "kube-api-access-qzmzb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:47:54.434846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.434745    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:47:54.435729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435703    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5d3dab08-fc99-49f3-b53b-8cf31ef9950e" containerName="revoke-keys"
Oct 30 22:47:54.435729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435729    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d3dab08-fc99-49f3-b53b-8cf31ef9950e" containerName="revoke-keys"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435781    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" containerName="payload-processing"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435790    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" containerName="payload-processing"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435807    4882 cpu_manager.go:414] "containerMap: removing container" podUID="e51de84d-55c4-43ab-bcfb-9709d87e7753" containerName="payload-pre-processing"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435816    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e51de84d-55c4-43ab-bcfb-9709d87e7753" containerName="payload-pre-processing"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435834    4882 cpu_manager.go:414] "containerMap: removing container" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy"
Oct 30 22:47:54.435945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435841    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy"
Oct 30 22:47:54.436261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435969    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d3dab08-fc99-49f3-b53b-8cf31ef9950e" containerName="revoke-keys"
Oct 30 22:47:54.436261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435989    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" containerName="payload-processing"
Oct 30 22:47:54.436261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.435999    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="e51de84d-55c4-43ab-bcfb-9709d87e7753" containerName="payload-pre-processing"
Oct 30 22:47:54.436261 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.436012    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="71a90030-8be8-44df-a758-a6e886b7123b" containerName="istio-proxy"
Oct 30 22:47:54.440104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.440079    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.440311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.440280    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzmzb\" (UniqueName: \"kubernetes.io/projected/e51de84d-55c4-43ab-bcfb-9709d87e7753-kube-api-access-qzmzb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:47:54.507625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.504573    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:47:54.507625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.505036    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-extmod-a-0c4786-openshift-default-dockercfg-6485q\""
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543332    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lwkh\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543397    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543442    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543505    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543559    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543600    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543702    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543744    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.546173 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.543808    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.551851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.551810    4882 generic.go:356] "Generic (PLEG): container finished" podID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" containerID="e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200" exitCode=0
Oct 30 22:47:54.552013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.551856    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"
Oct 30 22:47:54.552111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.552004    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" event={"ID":"c0755c3c-e2a0-44e1-af29-df19118a6ef4","Type":"ContainerDied","Data":"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"}
Oct 30 22:47:54.552111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.552056    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj" event={"ID":"c0755c3c-e2a0-44e1-af29-df19118a6ef4","Type":"ContainerDied","Data":"efb88ef11685ad0754cd865a830178f40185a566766852d2ed4f1d59ba5388c4"}
Oct 30 22:47:54.552111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.552087    4882 scope.go:122] "RemoveContainer" containerID="e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"
Oct 30 22:47:54.554669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.554639    4882 generic.go:356] "Generic (PLEG): container finished" podID="e51de84d-55c4-43ab-bcfb-9709d87e7753" containerID="d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202" exitCode=0
Oct 30 22:47:54.554669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.554709    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" event={"ID":"e51de84d-55c4-43ab-bcfb-9709d87e7753","Type":"ContainerDied","Data":"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"}
Oct 30 22:47:54.554669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.554722    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"
Oct 30 22:47:54.554669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.554785    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs" event={"ID":"e51de84d-55c4-43ab-bcfb-9709d87e7753","Type":"ContainerDied","Data":"cb7224f3a6201d1aaaf00da3bff3ee197154bc21d5ad9d43b6a0f4d5d379e752"}
Oct 30 22:47:54.570518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.570374    4882 scope.go:122] "RemoveContainer" containerID="e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"
Oct 30 22:47:54.572128 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:54.571634    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200\": container with ID starting with e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200 not found: ID does not exist" containerID="e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"
Oct 30 22:47:54.572128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.571675    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200"} err="failed to get container status \"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200\": rpc error: code = NotFound desc = could not find container \"e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200\": container with ID starting with e25c106a88fb8bfdb9bed2f8e15225a79a9ce826bc04cc384dedbab83c90a200 not found: ID does not exist"
Oct 30 22:47:54.572128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.571703    4882 scope.go:122] "RemoveContainer" containerID="d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"
Oct 30 22:47:54.585867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.585842    4882 scope.go:122] "RemoveContainer" containerID="d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"
Oct 30 22:47:54.586171 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:47:54.586149    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202\": container with ID starting with d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202 not found: ID does not exist" containerID="d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"
Oct 30 22:47:54.586248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.586179    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202"} err="failed to get container status \"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202\": rpc error: code = NotFound desc = could not find container \"d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202\": container with ID starting with d6f067f0a3727243b62323a78468865452c7f23e9bf1e0af0dad08abf484d202 not found: ID does not exist"
Oct 30 22:47:54.644886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.644846    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.644920    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.644991    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645051    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645077    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645175    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645209    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645270    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645309    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lwkh\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645426    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645750    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645844    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645890    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.645984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.645893    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.647414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.647392    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.647742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.647723    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.690486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.687775    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:47:54.691137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.691104    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-1a5f4a-77d955759f-wrsjj"]
Oct 30 22:47:54.707151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.707122    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.707368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.707345    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lwkh\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\") pod \"e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") " pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.754895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.754861    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:54.859661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.859625    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:47:54.865019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.864986    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-1a5f4a-64b968b555-4bmjs"]
Oct 30 22:47:54.945466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.945370    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0755c3c-e2a0-44e1-af29-df19118a6ef4" path="/var/lib/kubelet/pods/c0755c3c-e2a0-44e1-af29-df19118a6ef4/volumes"
Oct 30 22:47:54.946053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:54.946020    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e51de84d-55c4-43ab-bcfb-9709d87e7753" path="/var/lib/kubelet/pods/e51de84d-55c4-43ab-bcfb-9709d87e7753/volumes"
Oct 30 22:47:55.188949 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:47:55.188904    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f04c653_84be_4ae7_94c7_99a00323d294.slice/crio-bf5eed93ec7b75aad6cea329dddad7759c7c68f4edb4ac98583880c6b2fa0861 WatchSource:0}: Error finding container bf5eed93ec7b75aad6cea329dddad7759c7c68f4edb4ac98583880c6b2fa0861: Status 404 returned error can't find the container with id bf5eed93ec7b75aad6cea329dddad7759c7c68f4edb4ac98583880c6b2fa0861
Oct 30 22:47:55.190892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.190851    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:47:55.191827 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.191791    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:47:55.191940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.191863    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:47:55.191940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.191898    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:47:55.567937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.567834    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerStarted","Data":"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"}
Oct 30 22:47:55.567937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.567929    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerStarted","Data":"bf5eed93ec7b75aad6cea329dddad7759c7c68f4edb4ac98583880c6b2fa0861"}
Oct 30 22:47:55.692220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.692160    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podStartSLOduration=1.692143264 podStartE2EDuration="1.692143264s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:47:55.689032882 +0000 UTC m=+4198.862357309" watchObservedRunningTime="2026-10-30 22:47:55.692143264 +0000 UTC m=+4198.865467690"
Oct 30 22:47:55.755377 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:55.755344    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:56.760950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:56.760918    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:57.582770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:57.582735    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:57.583780 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:57.583754    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:47:57.585306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:57.585286    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:47:57.607518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:47:57.607493    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:48:00.307297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:00.307253    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:00.307297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:00.307267    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:48:00.312198 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:00.312170    4882 scope.go:122] "RemoveContainer" containerID="bcfa644d3aae00e5b13d35f6287ea87cdd8a83cd3aa9a81f23f55fc9a54118b5"
Oct 30 22:48:00.322118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:00.322098    4882 scope.go:122] "RemoveContainer" containerID="c9329fe8b595deaf3fb5d70684aa01b6d34b3c1e2660ea9f7e4976c8c7e84940"
Oct 30 22:48:02.673660 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:02.673624    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"]
Oct 30 22:48:02.723782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:02.723749    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w1-323082-79zzl"]
Oct 30 22:48:02.943617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:02.943528    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12caa3cc-3532-4a6d-9d45-e1dd9d94d13d" path="/var/lib/kubelet/pods/12caa3cc-3532-4a6d-9d45-e1dd9d94d13d/volumes"
Oct 30 22:48:04.855786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.855747    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:48:04.859755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.859734    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:04.864773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.864748    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:04.864935 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.864913    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vqhs\" (UniqueName: \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:04.919660 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.919620    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:48:04.963184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.963147    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-a-0c4786\""
Oct 30 22:48:04.963184 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.963147    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-a-0c4786-dockercfg-kpjpd\""
Oct 30 22:48:04.966039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.966014    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vqhs\" (UniqueName: \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:04.966152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.966133    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:04.966936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:04.966907    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:05.017009 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.016972    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:48:05.021332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.021307    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.030411 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.030376    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vqhs\" (UniqueName: \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\") pod \"payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:05.067555 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.067517    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mw84d\" (UniqueName: \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.067768 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.067686    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.078720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.078684    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:48:05.168382 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.168284    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mw84d\" (UniqueName: \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.168574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.168400    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.169142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.169116    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.174071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.174048    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:05.230899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.230868    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mw84d\" (UniqueName: \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\") pod \"payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") " pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.348851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.348814    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:05.547836 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:05.547803    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b6051e_995a_460c_b8f6_07592f9d6846.slice/crio-b99f07bc5facc71bbc92d9a9530d5918fa98b1a1df8ed7e65d1e37b8ef0cc0ce WatchSource:0}: Error finding container b99f07bc5facc71bbc92d9a9530d5918fa98b1a1df8ed7e65d1e37b8ef0cc0ce: Status 404 returned error can't find the container with id b99f07bc5facc71bbc92d9a9530d5918fa98b1a1df8ed7e65d1e37b8ef0cc0ce
Oct 30 22:48:05.550541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.550516    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:48:05.638005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.637974    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" event={"ID":"42b6051e-995a-460c-b8f6-07592f9d6846","Type":"ContainerStarted","Data":"b99f07bc5facc71bbc92d9a9530d5918fa98b1a1df8ed7e65d1e37b8ef0cc0ce"}
Oct 30 22:48:05.726032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.725997    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:48:05.741625 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:05.741578    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87fc4f9_8c7b_424b_b31a_67935563bec0.slice/crio-a22c781fab4c5bb0490d3012c60304e7a77d0890a2d855a24668f71d84a90df2 WatchSource:0}: Error finding container a22c781fab4c5bb0490d3012c60304e7a77d0890a2d855a24668f71d84a90df2: Status 404 returned error can't find the container with id a22c781fab4c5bb0490d3012c60304e7a77d0890a2d855a24668f71d84a90df2
Oct 30 22:48:05.860046 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.860017    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:48:05.874834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.874808    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\") pod \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") "
Oct 30 22:48:05.875201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.875022    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\" (UniqueName: \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\") pod \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\" (UID: \"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6\") "
Oct 30 22:48:05.875340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.875313    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume" pod "794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" (UID: "794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:05.877429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.877401    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh" pod "794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" (UID: "794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6"). InnerVolumeSpecName "kube-api-access-r2xfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:05.954339 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.954310    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:48:05.975958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.975922    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\" (UniqueName: \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\") pod \"9c8f639f-422a-431e-b36a-61e49aa4c618\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") "
Oct 30 22:48:05.976147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.976035    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\") pod \"9c8f639f-422a-431e-b36a-61e49aa4c618\" (UID: \"9c8f639f-422a-431e-b36a-61e49aa4c618\") "
Oct 30 22:48:05.976395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.976360    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume" pod "9c8f639f-422a-431e-b36a-61e49aa4c618" (UID: "9c8f639f-422a-431e-b36a-61e49aa4c618"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:05.976557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.976527    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c8f639f-422a-431e-b36a-61e49aa4c618-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:05.976557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.976551    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2xfh\" (UniqueName: \"kubernetes.io/projected/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-kube-api-access-r2xfh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:05.976738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.976567    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:05.978674 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:05.978642    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl" pod "9c8f639f-422a-431e-b36a-61e49aa4c618" (UID: "9c8f639f-422a-431e-b36a-61e49aa4c618"). InnerVolumeSpecName "kube-api-access-8srvl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:06.078041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.077969    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srvl\" (UniqueName: \"kubernetes.io/projected/9c8f639f-422a-431e-b36a-61e49aa4c618-kube-api-access-8srvl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:06.646669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.646625    4882 generic.go:356] "Generic (PLEG): container finished" podID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerID="d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e" exitCode=0
Oct 30 22:48:06.646669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.646660    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"
Oct 30 22:48:06.646934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.646697    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" event={"ID":"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6","Type":"ContainerDied","Data":"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"}
Oct 30 22:48:06.646934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.646757    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx" event={"ID":"794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6","Type":"ContainerDied","Data":"d0bf4c6c442417fba5386b8a5f52cc87522b99d14bba48d6310f7487f638fda9"}
Oct 30 22:48:06.646934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.646789    4882 scope.go:122] "RemoveContainer" containerID="d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"
Oct 30 22:48:06.648881 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.648841    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" event={"ID":"42b6051e-995a-460c-b8f6-07592f9d6846","Type":"ContainerStarted","Data":"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"}
Oct 30 22:48:06.649051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.648994    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:06.650403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.650376    4882 generic.go:356] "Generic (PLEG): container finished" podID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerID="b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921" exitCode=0
Oct 30 22:48:06.650527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.650412    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"
Oct 30 22:48:06.650527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.650452    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" event={"ID":"9c8f639f-422a-431e-b36a-61e49aa4c618","Type":"ContainerDied","Data":"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"}
Oct 30 22:48:06.650527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.650502    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx" event={"ID":"9c8f639f-422a-431e-b36a-61e49aa4c618","Type":"ContainerDied","Data":"5bc8757aed6d82dcd7a7dca717d7cbeb12289041e454e45d2ec34ea5e3dd6a63"}
Oct 30 22:48:06.652651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.652630    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" event={"ID":"d87fc4f9-8c7b-424b-b31a-67935563bec0","Type":"ContainerStarted","Data":"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"}
Oct 30 22:48:06.652807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.652675    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" event={"ID":"d87fc4f9-8c7b-424b-b31a-67935563bec0","Type":"ContainerStarted","Data":"a22c781fab4c5bb0490d3012c60304e7a77d0890a2d855a24668f71d84a90df2"}
Oct 30 22:48:06.652972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.652946    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:06.660628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.660610    4882 scope.go:122] "RemoveContainer" containerID="d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"
Oct 30 22:48:06.660915 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:06.660878    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e\": container with ID starting with d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e not found: ID does not exist" containerID="d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"
Oct 30 22:48:06.661010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.660921    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e"} err="failed to get container status \"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e\": rpc error: code = NotFound desc = could not find container \"d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e\": container with ID starting with d9147331680c49025de58cdd710e64395830a1e17cb7085cc707574c130c3c4e not found: ID does not exist"
Oct 30 22:48:06.661010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.660949    4882 scope.go:122] "RemoveContainer" containerID="b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"
Oct 30 22:48:06.671016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.670996    4882 scope.go:122] "RemoveContainer" containerID="b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"
Oct 30 22:48:06.671296 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:06.671278    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921\": container with ID starting with b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921 not found: ID does not exist" containerID="b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"
Oct 30 22:48:06.671348 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.671304    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921"} err="failed to get container status \"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921\": rpc error: code = NotFound desc = could not find container \"b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921\": container with ID starting with b7dcd5dc7c222af87ba76efc3da479238eed3f075ab1044468ab44e2f2764921 not found: ID does not exist"
Oct 30 22:48:06.817684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.817632    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" podStartSLOduration=2.817618329 podStartE2EDuration="2.817618329s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:06.81673638 +0000 UTC m=+4209.990060809" watchObservedRunningTime="2026-10-30 22:48:06.817618329 +0000 UTC m=+4209.990942750"
Oct 30 22:48:06.933309 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.933227    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:48:06.944705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:06.944674    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w1-323082-c7fb45f99-bcvtx"]
Oct 30 22:48:07.052781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:07.052725    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" podStartSLOduration=3.052710489 podStartE2EDuration="3.052710489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:07.050360023 +0000 UTC m=+4210.223684449" watchObservedRunningTime="2026-10-30 22:48:07.052710489 +0000 UTC m=+4210.226034937"
Oct 30 22:48:07.165761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:07.165723    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:48:07.169491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:07.169461    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w1-323082-d7cdd9b7c-nrkfx"]
Oct 30 22:48:08.943179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:08.943141    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" path="/var/lib/kubelet/pods/794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6/volumes"
Oct 30 22:48:08.943616 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:08.943564    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" path="/var/lib/kubelet/pods/9c8f639f-422a-431e-b36a-61e49aa4c618/volumes"
Oct 30 22:48:10.800067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800027    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"]
Oct 30 22:48:10.800684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800666    4882 cpu_manager.go:414] "containerMap: removing container" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing"
Oct 30 22:48:10.800684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800684    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing"
Oct 30 22:48:10.800795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800710    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing"
Oct 30 22:48:10.800795 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800716    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing"
Oct 30 22:48:10.800871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800825    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c8f639f-422a-431e-b36a-61e49aa4c618" containerName="payload-pre-processing"
Oct 30 22:48:10.800871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.800840    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="794ee7b6-6c2e-4821-8edf-e3c3f8b1cab6" containerName="payload-processing"
Oct 30 22:48:10.805789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.805763    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.828015 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.827983    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.828168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.828121    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crmzx\" (UniqueName: \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.828212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.828185    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.857580 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.857549    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"]
Oct 30 22:48:10.904700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.904669    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:10.904700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.904701    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:48:10.929268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.929236    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.929492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.929340    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crmzx\" (UniqueName: \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.929492 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.929378    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.929919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.929895    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.931882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.931851    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:10.991277 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:10.991242    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crmzx\" (UniqueName: \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\") pod \"maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:11.119385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:11.119340    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:11.497227 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:11.497192    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506ad38e_9913_490f_b454_8742653ba30e.slice/crio-37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a WatchSource:0}: Error finding container 37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a: Status 404 returned error can't find the container with id 37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a
Oct 30 22:48:11.499541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:11.499506    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"]
Oct 30 22:48:11.686803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:11.686756    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn" event={"ID":"506ad38e-9913-490f-b454-8742653ba30e","Type":"ContainerStarted","Data":"37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a"}
Oct 30 22:48:12.694889 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:12.694804    4882 generic.go:356] "Generic (PLEG): container finished" podID="506ad38e-9913-490f-b454-8742653ba30e" containerID="4ff9e876ba4d218fc63096fe0cf0ab4d7404cb822d3131a1c65520af2b0980da" exitCode=0
Oct 30 22:48:12.695273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:12.694883    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn" event={"ID":"506ad38e-9913-490f-b454-8742653ba30e","Type":"ContainerDied","Data":"4ff9e876ba4d218fc63096fe0cf0ab4d7404cb822d3131a1c65520af2b0980da"}
Oct 30 22:48:13.856033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.856006    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:13.965166 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.965073    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\") pod \"506ad38e-9913-490f-b454-8742653ba30e\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") "
Oct 30 22:48:13.965372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.965228    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\" (UniqueName: \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\") pod \"506ad38e-9913-490f-b454-8742653ba30e\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") "
Oct 30 22:48:13.965372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.965295    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\") pod \"506ad38e-9913-490f-b454-8742653ba30e\" (UID: \"506ad38e-9913-490f-b454-8742653ba30e\") "
Oct 30 22:48:13.965701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.965672    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca" pod "506ad38e-9913-490f-b454-8742653ba30e" (UID: "506ad38e-9913-490f-b454-8742653ba30e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:13.967343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.967319    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx" pod "506ad38e-9913-490f-b454-8742653ba30e" (UID: "506ad38e-9913-490f-b454-8742653ba30e"). InnerVolumeSpecName "kube-api-access-crmzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:13.967450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:13.967399    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca" pod "506ad38e-9913-490f-b454-8742653ba30e" (UID: "506ad38e-9913-490f-b454-8742653ba30e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:14.066605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.066558    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/506ad38e-9913-490f-b454-8742653ba30e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.066605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.066610    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/506ad38e-9913-490f-b454-8742653ba30e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.066841 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.066622    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crmzx\" (UniqueName: \"kubernetes.io/projected/506ad38e-9913-490f-b454-8742653ba30e-kube-api-access-crmzx\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:14.710135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.710093    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"
Oct 30 22:48:14.710135 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.710125    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn" event={"ID":"506ad38e-9913-490f-b454-8742653ba30e","Type":"ContainerDied","Data":"37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a"}
Oct 30 22:48:14.710361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:14.710204    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="37ee5f270856fce5f05aecbf6e7c27436d3d89887d07fc4bf3bbdb6dabe7e42a"
Oct 30 22:48:15.187279 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:15.187240    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:48:15.212450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:15.212424    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 22:48:17.664330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:17.664300    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:48:17.664801 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:17.664465    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:48:20.808732 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:20.808694    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:48:28.125056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:28.125024    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"]
Oct 30 22:48:28.175120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:28.175083    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-8c852b-6ppl6"]
Oct 30 22:48:28.943147 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:28.943098    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0a3ed52-323d-4f32-840d-5ea6f9accd6a" path="/var/lib/kubelet/pods/e0a3ed52-323d-4f32-840d-5ea6f9accd6a/volumes"
Oct 30 22:48:30.208641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.208580    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" podUID="59948204-a285-4476-8079-17089fdddef9" containerName="main" containerID="cri-o://ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140" gracePeriod=45
Oct 30 22:48:30.235140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.235104    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" containerID="cri-o://ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd" gracePeriod=45
Oct 30 22:48:30.340243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.340194    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main" probeResult="failure" output="Get \"https://10.132.0.49:8000/ready\": dial tcp 10.132.0.49:8000: connect: connection refused"
Oct 30 22:48:30.518645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.518615    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:48:30.522021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.521999    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 22:48:30.652554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652514    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.652765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652559    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.652765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652606    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.652765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652644    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652910    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.652925    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653080    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\" (UniqueName: \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653151    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\" (UniqueName: \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653227    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653312    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653411    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653487    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\") pod \"59948204-a285-4476-8079-17089fdddef9\" (UID: \"59948204-a285-4476-8079-17089fdddef9\") "
Oct 30 22:48:30.653630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653572    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.654185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.653657    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\" (UniqueName: \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\") pod \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\" (UID: \"37a3fbe2-9e70-49d9-a40c-c4b22aa90304\") "
Oct 30 22:48:30.654252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.654219    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.654252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.654241    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.655371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655338    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.655371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655359    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "kube-api-access-krppg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:30.655718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655541    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.655718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655619    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.655718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655628    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.656078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.655938    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.657802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.657776    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "kube-api-access-blkr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:30.658540 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.658514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:30.658826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.658803    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs" pod "59948204-a285-4476-8079-17089fdddef9" (UID: "59948204-a285-4476-8079-17089fdddef9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:30.660576 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.660391    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm" pod "37a3fbe2-9e70-49d9-a40c-c4b22aa90304" (UID: "37a3fbe2-9e70-49d9-a40c-c4b22aa90304"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755150    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krppg\" (UniqueName: \"kubernetes.io/projected/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-kube-api-access-krppg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755186    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755198    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blkr5\" (UniqueName: \"kubernetes.io/projected/59948204-a285-4476-8079-17089fdddef9-kube-api-access-blkr5\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755209    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/59948204-a285-4476-8079-17089fdddef9-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755223    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755234    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.755248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755246    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.756252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755263    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/59948204-a285-4476-8079-17089fdddef9-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.756252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755278    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.756252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.755289    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/37a3fbe2-9e70-49d9-a40c-c4b22aa90304-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:30.836038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.836006    4882 generic.go:356] "Generic (PLEG): container finished" podID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerID="ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd" exitCode=0
Oct 30 22:48:30.836234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.836058    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"
Oct 30 22:48:30.836234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.836089    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" event={"ID":"37a3fbe2-9e70-49d9-a40c-c4b22aa90304","Type":"ContainerDied","Data":"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"}
Oct 30 22:48:30.836234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.836136    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx" event={"ID":"37a3fbe2-9e70-49d9-a40c-c4b22aa90304","Type":"ContainerDied","Data":"a63662a0b099c3efc4c2ebcd510fec671f8c508315684bb11d1205ca5fcf7871"}
Oct 30 22:48:30.836234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.836159    4882 scope.go:122] "RemoveContainer" containerID="ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"
Oct 30 22:48:30.837707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.837679    4882 generic.go:356] "Generic (PLEG): container finished" podID="59948204-a285-4476-8079-17089fdddef9" containerID="ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140" exitCode=0
Oct 30 22:48:30.837851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.837714    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"
Oct 30 22:48:30.837851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.837754    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" event={"ID":"59948204-a285-4476-8079-17089fdddef9","Type":"ContainerDied","Data":"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"}
Oct 30 22:48:30.837851 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.837794    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr" event={"ID":"59948204-a285-4476-8079-17089fdddef9","Type":"ContainerDied","Data":"613e046b3c5ffe506771dc3b0ffdbc6978249b4a404f0e3d88bd5f81b9e6b4eb"}
Oct 30 22:48:30.847496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.847475    4882 scope.go:122] "RemoveContainer" containerID="ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"
Oct 30 22:48:30.847863 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:30.847836    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd\": container with ID starting with ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd not found: ID does not exist" containerID="ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"
Oct 30 22:48:30.847973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.847870    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd"} err="failed to get container status \"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd\": rpc error: code = NotFound desc = could not find container \"ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd\": container with ID starting with ad21bfb57724282ca0405d90f5533aad45ea8bc14827a61559b3cd5eea0e8ebd not found: ID does not exist"
Oct 30 22:48:30.847973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.847894    4882 scope.go:122] "RemoveContainer" containerID="ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"
Oct 30 22:48:30.857353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.857336    4882 scope.go:122] "RemoveContainer" containerID="ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"
Oct 30 22:48:30.857681 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:30.857653    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140\": container with ID starting with ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140 not found: ID does not exist" containerID="ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"
Oct 30 22:48:30.857787 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:30.857686    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140"} err="failed to get container status \"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140\": rpc error: code = NotFound desc = could not find container \"ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140\": container with ID starting with ef0155b80a5b43d81244af25a6a8deda6091daae42fae2222e6721ad21c00140 not found: ID does not exist"
Oct 30 22:48:31.002979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:31.002945    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 22:48:31.007056 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:31.006978    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/simulator-w1-3c4802-kserve-75bcf5d784-l7szx"]
Oct 30 22:48:31.118386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:31.118350    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:48:31.121922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:31.121895    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-3c4802/distinct-2-w1-3c4802-kserve-5cdc745f79-j94pr"]
Oct 30 22:48:32.946037 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:32.945998    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" path="/var/lib/kubelet/pods/37a3fbe2-9e70-49d9-a40c-c4b22aa90304/volumes"
Oct 30 22:48:32.946518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:32.946456    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59948204-a285-4476-8079-17089fdddef9" path="/var/lib/kubelet/pods/59948204-a285-4476-8079-17089fdddef9/volumes"
Oct 30 22:48:34.080961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:34.080877    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"]
Oct 30 22:48:34.131105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:34.131072    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-03e614-fktnf"]
Oct 30 22:48:34.943530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:34.943492    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d3dab08-fc99-49f3-b53b-8cf31ef9950e" path="/var/lib/kubelet/pods/5d3dab08-fc99-49f3-b53b-8cf31ef9950e/volumes"
Oct 30 22:48:36.115700 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.115663    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"]
Oct 30 22:48:36.116254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116239    4882 cpu_manager.go:414] "containerMap: removing container" podUID="506ad38e-9913-490f-b454-8742653ba30e" containerName="revoke-keys"
Oct 30 22:48:36.116254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116253    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506ad38e-9913-490f-b454-8742653ba30e" containerName="revoke-keys"
Oct 30 22:48:36.116388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116272    4882 cpu_manager.go:414] "containerMap: removing container" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main"
Oct 30 22:48:36.116388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116277    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main"
Oct 30 22:48:36.116388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116306    4882 cpu_manager.go:414] "containerMap: removing container" podUID="59948204-a285-4476-8079-17089fdddef9" containerName="main"
Oct 30 22:48:36.116388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116311    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59948204-a285-4476-8079-17089fdddef9" containerName="main"
Oct 30 22:48:36.116388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116389    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="59948204-a285-4476-8079-17089fdddef9" containerName="main"
Oct 30 22:48:36.116564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116403    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="506ad38e-9913-490f-b454-8742653ba30e" containerName="revoke-keys"
Oct 30 22:48:36.116564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.116411    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="37a3fbe2-9e70-49d9-a40c-c4b22aa90304" containerName="main"
Oct 30 22:48:36.120142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.120122    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.174203 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.174169    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"]
Oct 30 22:48:36.217451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.217404    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.217669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.217559    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.217753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.217733    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5nhd\" (UniqueName: \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.221101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.221080    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:48:36.221101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.221089    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:48:36.319076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.319031    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5nhd\" (UniqueName: \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.319274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.319092    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.319274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.319148    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.319854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.319832    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.321574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.321551    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.380645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.380563    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5nhd\" (UniqueName: \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\") pod \"maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.433679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.433644    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:36.813727 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:36.813685    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8f69f93_b308_4673_889c_e3d56374e2a8.slice/crio-8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb WatchSource:0}: Error finding container 8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb: Status 404 returned error can't find the container with id 8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb
Oct 30 22:48:36.815419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.815386    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"]
Oct 30 22:48:36.880337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:36.880300    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc" event={"ID":"f8f69f93-b308-4673-889c-e3d56374e2a8","Type":"ContainerStarted","Data":"8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb"}
Oct 30 22:48:37.888006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:37.887973    4882 generic.go:356] "Generic (PLEG): container finished" podID="f8f69f93-b308-4673-889c-e3d56374e2a8" containerID="8cc1c3776fd43acf51882fe8ab64a2fce9b119ea5bfb1e59b52b7699842e3052" exitCode=0
Oct 30 22:48:37.888425 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:37.888051    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc" event={"ID":"f8f69f93-b308-4673-889c-e3d56374e2a8","Type":"ContainerDied","Data":"8cc1c3776fd43acf51882fe8ab64a2fce9b119ea5bfb1e59b52b7699842e3052"}
Oct 30 22:48:38.057876 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.057781    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"]
Oct 30 22:48:38.061987 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.061962    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.101201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.101171    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"]
Oct 30 22:48:38.105409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.105378    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.112409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.112376    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"]
Oct 30 22:48:38.142429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142394    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z54kl\" (UniqueName: \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.142575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142449    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k887x\" (UniqueName: \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.142575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142537    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.142753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142643    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.142753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142741    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.142936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.142769    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.159834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.159800    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"]
Oct 30 22:48:38.243393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243346    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z54kl\" (UniqueName: \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.243393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243395    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k887x\" (UniqueName: \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.243722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243426    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.243722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243490    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.243722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243534    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.243722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.243553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.244260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.244226    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.244260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.244252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.246220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.246201    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.246294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.246227    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.308031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.307960    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z54kl\" (UniqueName: \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\") pod \"maas-api-revoke-sub-a6ad0560-tvckc\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.308175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.308122    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k887x\" (UniqueName: \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\") pod \"maas-api-revoke-sub-459ad77a-kfnrk\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.376546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.376508    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:38.421902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.421866    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:38.752104 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:38.752060    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcb46564_1dff_4f59_9ad1_8ff2e09fb0e6.slice/crio-5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4 WatchSource:0}: Error finding container 5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4: Status 404 returned error can't find the container with id 5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4
Oct 30 22:48:38.756159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.756135    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"]
Oct 30 22:48:38.812410 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:38.812381    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9b140d6_8a80_4af2_815e_da7355134862.slice/crio-e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd WatchSource:0}: Error finding container e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd: Status 404 returned error can't find the container with id e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd
Oct 30 22:48:38.814124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.814091    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"]
Oct 30 22:48:38.895808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.895767    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc" event={"ID":"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6","Type":"ContainerStarted","Data":"5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4"}
Oct 30 22:48:38.897087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:38.897059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk" event={"ID":"b9b140d6-8a80-4af2-815e-da7355134862","Type":"ContainerStarted","Data":"e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd"}
Oct 30 22:48:39.037115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.037085    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:39.153335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.153300    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\") pod \"f8f69f93-b308-4673-889c-e3d56374e2a8\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") "
Oct 30 22:48:39.153495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.153364    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\") pod \"f8f69f93-b308-4673-889c-e3d56374e2a8\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") "
Oct 30 22:48:39.153495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.153417    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\" (UniqueName: \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\") pod \"f8f69f93-b308-4673-889c-e3d56374e2a8\" (UID: \"f8f69f93-b308-4673-889c-e3d56374e2a8\") "
Oct 30 22:48:39.153810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.153780    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca" pod "f8f69f93-b308-4673-889c-e3d56374e2a8" (UID: "f8f69f93-b308-4673-889c-e3d56374e2a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:39.154224 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.154062    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:39.156643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.156579    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca" pod "f8f69f93-b308-4673-889c-e3d56374e2a8" (UID: "f8f69f93-b308-4673-889c-e3d56374e2a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:39.156643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.156638    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd" pod "f8f69f93-b308-4673-889c-e3d56374e2a8" (UID: "f8f69f93-b308-4673-889c-e3d56374e2a8"). InnerVolumeSpecName "kube-api-access-s5nhd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:39.255491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.255459    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f8f69f93-b308-4673-889c-e3d56374e2a8-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:39.255491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.255490    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5nhd\" (UniqueName: \"kubernetes.io/projected/f8f69f93-b308-4673-889c-e3d56374e2a8-kube-api-access-s5nhd\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:39.905809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.905773    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"
Oct 30 22:48:39.905809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.905803    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc" event={"ID":"f8f69f93-b308-4673-889c-e3d56374e2a8","Type":"ContainerDied","Data":"8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb"}
Oct 30 22:48:39.906328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.905861    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc5f571a67d36d0d2d7adbb531b9605d4585e7243061bd624b292c884d2dcdb"
Oct 30 22:48:39.907656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.907620    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk" event={"ID":"b9b140d6-8a80-4af2-815e-da7355134862","Type":"ContainerStarted","Data":"f60f513e5c6f121fd0c585ed96781df06572ea763bd670583e17c8b6b1528360"}
Oct 30 22:48:39.909457 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.909431    4882 generic.go:356] "Generic (PLEG): container finished" podID="dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" containerID="8cab88ec6786f370375145f29438cd48976d61f6d01c2d9bb233c3913dbd201d" exitCode=0
Oct 30 22:48:39.909632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:39.909485    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc" event={"ID":"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6","Type":"ContainerDied","Data":"8cab88ec6786f370375145f29438cd48976d61f6d01c2d9bb233c3913dbd201d"}
Oct 30 22:48:40.029798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:40.029688    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk" podStartSLOduration=2.029674277 podStartE2EDuration="2.029674277s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:48:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:48:40.027199223 +0000 UTC m=+4243.200523652" watchObservedRunningTime="2026-10-30 22:48:40.029674277 +0000 UTC m=+4243.202998766"
Oct 30 22:48:40.920810 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:40.920770    4882 generic.go:356] "Generic (PLEG): container finished" podID="b9b140d6-8a80-4af2-815e-da7355134862" containerID="f60f513e5c6f121fd0c585ed96781df06572ea763bd670583e17c8b6b1528360" exitCode=0
Oct 30 22:48:40.921376 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:40.920928    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk" event={"ID":"b9b140d6-8a80-4af2-815e-da7355134862","Type":"ContainerDied","Data":"f60f513e5c6f121fd0c585ed96781df06572ea763bd670583e17c8b6b1528360"}
Oct 30 22:48:41.070877 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.070844    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:41.176802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.176722    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\") pod \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") "
Oct 30 22:48:41.176986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.176868    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\" (UniqueName: \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\") pod \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") "
Oct 30 22:48:41.176986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.176946    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\") pod \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\" (UID: \"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6\") "
Oct 30 22:48:41.177545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.177512    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca" pod "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" (UID: "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:41.179755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.179724    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca" pod "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" (UID: "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:41.181095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.181063    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl" pod "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" (UID: "dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6"). InnerVolumeSpecName "kube-api-access-z54kl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:41.278410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.278370    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z54kl\" (UniqueName: \"kubernetes.io/projected/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-kube-api-access-z54kl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:41.278410 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.278411    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:41.278636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.278428    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:41.928631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.928578    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"
Oct 30 22:48:41.928631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.928613    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc" event={"ID":"dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6","Type":"ContainerDied","Data":"5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4"}
Oct 30 22:48:41.929206 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:41.928660    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5602127b24247c0e8d0d9d47308afcda5abfceb8e1414888e4ed7b9e56f2d4c4"
Oct 30 22:48:42.068067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068033    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"]
Oct 30 22:48:42.068702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068685    4882 cpu_manager.go:414] "containerMap: removing container" podUID="dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" containerName="revoke-keys"
Oct 30 22:48:42.068759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068703    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" containerName="revoke-keys"
Oct 30 22:48:42.068759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068731    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f8f69f93-b308-4673-889c-e3d56374e2a8" containerName="revoke-keys"
Oct 30 22:48:42.068759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068739    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8f69f93-b308-4673-889c-e3d56374e2a8" containerName="revoke-keys"
Oct 30 22:48:42.068859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068821    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8f69f93-b308-4673-889c-e3d56374e2a8" containerName="revoke-keys"
Oct 30 22:48:42.068859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.068837    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" containerName="revoke-keys"
Oct 30 22:48:42.072468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.072452    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.082242 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.082219    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:42.126101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.126067    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"]
Oct 30 22:48:42.189301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189211    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\") pod \"b9b140d6-8a80-4af2-815e-da7355134862\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") "
Oct 30 22:48:42.189472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189448    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\" (UniqueName: \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\") pod \"b9b140d6-8a80-4af2-815e-da7355134862\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") "
Oct 30 22:48:42.189534 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189509    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\") pod \"b9b140d6-8a80-4af2-815e-da7355134862\" (UID: \"b9b140d6-8a80-4af2-815e-da7355134862\") "
Oct 30 22:48:42.189742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189709    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.189860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189793    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtfrq\" (UniqueName: \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.189930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189868    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.189991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.189958    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca" pod "b9b140d6-8a80-4af2-815e-da7355134862" (UID: "b9b140d6-8a80-4af2-815e-da7355134862"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:42.190220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.190196    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:42.191544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.191518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca" pod "b9b140d6-8a80-4af2-815e-da7355134862" (UID: "b9b140d6-8a80-4af2-815e-da7355134862"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:42.191721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.191705    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x" pod "b9b140d6-8a80-4af2-815e-da7355134862" (UID: "b9b140d6-8a80-4af2-815e-da7355134862"). InnerVolumeSpecName "kube-api-access-k887x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:42.291170 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291127    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.291360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291186    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtfrq\" (UniqueName: \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.291360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291222    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.291360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291331    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b9b140d6-8a80-4af2-815e-da7355134862-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:42.291360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291345    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k887x\" (UniqueName: \"kubernetes.io/projected/b9b140d6-8a80-4af2-815e-da7355134862-kube-api-access-k887x\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:42.292014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.291991    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.293647 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.293623    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.352958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.352922    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtfrq\" (UniqueName: \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.392000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.391964    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:42.769513 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:48:42.769480    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef76668a_6668_497f_8af0_e3d88796d54e.slice/crio-9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921 WatchSource:0}: Error finding container 9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921: Status 404 returned error can't find the container with id 9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921
Oct 30 22:48:42.772699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.772677    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"]
Oct 30 22:48:42.936635 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.936577    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"
Oct 30 22:48:42.946110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.946067    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk" event={"ID":"b9b140d6-8a80-4af2-815e-da7355134862","Type":"ContainerDied","Data":"e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd"}
Oct 30 22:48:42.946256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.946138    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e92faddab43dec3f8491420a3ba10544fd567dc91e90580ebd71bf3a514cc3fd"
Oct 30 22:48:42.946256 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:42.946157    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb" event={"ID":"ef76668a-6668-497f-8af0-e3d88796d54e","Type":"ContainerStarted","Data":"9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921"}
Oct 30 22:48:43.946249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:43.946166    4882 generic.go:356] "Generic (PLEG): container finished" podID="ef76668a-6668-497f-8af0-e3d88796d54e" containerID="9498325fdfbda0b7b0101f47dccf265e1ff94e2f8c55d3eb57a0dcee5ea67918" exitCode=0
Oct 30 22:48:43.946249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:43.946218    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb" event={"ID":"ef76668a-6668-497f-8af0-e3d88796d54e","Type":"ContainerDied","Data":"9498325fdfbda0b7b0101f47dccf265e1ff94e2f8c55d3eb57a0dcee5ea67918"}
Oct 30 22:48:45.112051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.112016    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:45.227033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.226936    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\" (UniqueName: \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\") pod \"ef76668a-6668-497f-8af0-e3d88796d54e\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") "
Oct 30 22:48:45.227033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.226991    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\") pod \"ef76668a-6668-497f-8af0-e3d88796d54e\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") "
Oct 30 22:48:45.227329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.227159    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\") pod \"ef76668a-6668-497f-8af0-e3d88796d54e\" (UID: \"ef76668a-6668-497f-8af0-e3d88796d54e\") "
Oct 30 22:48:45.227725 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.227692    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca" pod "ef76668a-6668-497f-8af0-e3d88796d54e" (UID: "ef76668a-6668-497f-8af0-e3d88796d54e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:45.229849 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.229813    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq" pod "ef76668a-6668-497f-8af0-e3d88796d54e" (UID: "ef76668a-6668-497f-8af0-e3d88796d54e"). InnerVolumeSpecName "kube-api-access-qtfrq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:45.230110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.230082    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca" pod "ef76668a-6668-497f-8af0-e3d88796d54e" (UID: "ef76668a-6668-497f-8af0-e3d88796d54e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:48:45.328945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.328905    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtfrq\" (UniqueName: \"kubernetes.io/projected/ef76668a-6668-497f-8af0-e3d88796d54e-kube-api-access-qtfrq\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:45.328945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.328941    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:45.328945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.328951    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef76668a-6668-497f-8af0-e3d88796d54e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:45.959802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.959768    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"
Oct 30 22:48:45.960027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.959806    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb" event={"ID":"ef76668a-6668-497f-8af0-e3d88796d54e","Type":"ContainerDied","Data":"9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921"}
Oct 30 22:48:45.960027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:45.959861    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9a8afa2825a9b828fb8c4687ec15b9a2c618c00c0666b8186f855baa73d8f921"
Oct 30 22:48:47.712430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:47.712394    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:48:47.821708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:47.821673    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:48:50.808672 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:50.808615    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:48:52.589220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:52.589180    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:48:52.696792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:52.696755    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:48:52.712963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:52.712925    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" podUID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" containerName="payload-processing" containerID="cri-o://1a72771695a05af9b29cf3391cfc2fb727325e8c502586e4b47a9857faca2cd5" gracePeriod=25
Oct 30 22:48:52.822833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:52.822639    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" podUID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" containerName="payload-pre-processing" containerID="cri-o://232d55f2aa3107cd13463fd49191c33296059150c9534cfed8c047cd9750d4b1" gracePeriod=25
Oct 30 22:48:53.017316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.017074    4882 generic.go:356] "Generic (PLEG): container finished" podID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" containerID="1a72771695a05af9b29cf3391cfc2fb727325e8c502586e4b47a9857faca2cd5" exitCode=0
Oct 30 22:48:53.017316 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.017249    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" event={"ID":"3c77ac22-68a0-46a7-b7f6-3e378e9874df","Type":"ContainerDied","Data":"1a72771695a05af9b29cf3391cfc2fb727325e8c502586e4b47a9857faca2cd5"}
Oct 30 22:48:53.019996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.019967    4882 generic.go:356] "Generic (PLEG): container finished" podID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" containerID="232d55f2aa3107cd13463fd49191c33296059150c9534cfed8c047cd9750d4b1" exitCode=0
Oct 30 22:48:53.020195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.020172    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" event={"ID":"9b0aed04-7985-4bd7-8602-ad9a1fdb8644","Type":"ContainerDied","Data":"232d55f2aa3107cd13463fd49191c33296059150c9534cfed8c047cd9750d4b1"}
Oct 30 22:48:53.076646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.076365    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:48:53.164274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.164245    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:48:53.219758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.219724    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\") pod \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") "
Oct 30 22:48:53.219980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.219800    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\" (UniqueName: \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\") pod \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\" (UID: \"3c77ac22-68a0-46a7-b7f6-3e378e9874df\") "
Oct 30 22:48:53.220234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.220203    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume" pod "3c77ac22-68a0-46a7-b7f6-3e378e9874df" (UID: "3c77ac22-68a0-46a7-b7f6-3e378e9874df"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:53.222109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.222081    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6" pod "3c77ac22-68a0-46a7-b7f6-3e378e9874df" (UID: "3c77ac22-68a0-46a7-b7f6-3e378e9874df"). InnerVolumeSpecName "kube-api-access-qr7g6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:53.321040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.320997    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\") pod \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") "
Oct 30 22:48:53.321227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.321107    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\" (UniqueName: \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\") pod \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\" (UID: \"9b0aed04-7985-4bd7-8602-ad9a1fdb8644\") "
Oct 30 22:48:53.321535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.321490    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume" pod "9b0aed04-7985-4bd7-8602-ad9a1fdb8644" (UID: "9b0aed04-7985-4bd7-8602-ad9a1fdb8644"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:53.321714 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.321690    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c77ac22-68a0-46a7-b7f6-3e378e9874df-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:53.321784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.321711    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr7g6\" (UniqueName: \"kubernetes.io/projected/3c77ac22-68a0-46a7-b7f6-3e378e9874df-kube-api-access-qr7g6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:53.321784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.321727    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:53.323997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.323960    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p" pod "9b0aed04-7985-4bd7-8602-ad9a1fdb8644" (UID: "9b0aed04-7985-4bd7-8602-ad9a1fdb8644"). InnerVolumeSpecName "kube-api-access-j825p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:53.423238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:53.423148    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j825p\" (UniqueName: \"kubernetes.io/projected/9b0aed04-7985-4bd7-8602-ad9a1fdb8644-kube-api-access-j825p\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:54.028062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.027968    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"
Oct 30 22:48:54.028062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.027999    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn" event={"ID":"3c77ac22-68a0-46a7-b7f6-3e378e9874df","Type":"ContainerDied","Data":"9273a5fca265e214549ab84b36215ca4ab4785024157dff5fc6d74d7cb63a059"}
Oct 30 22:48:54.028062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.028058    4882 scope.go:122] "RemoveContainer" containerID="1a72771695a05af9b29cf3391cfc2fb727325e8c502586e4b47a9857faca2cd5"
Oct 30 22:48:54.029634 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.029608    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4" event={"ID":"9b0aed04-7985-4bd7-8602-ad9a1fdb8644","Type":"ContainerDied","Data":"20df4cab038cec870ee8707adfc4e1d3375317f9ce4ac656fb3bc5a37366e9b9"}
Oct 30 22:48:54.029834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.029787    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"
Oct 30 22:48:54.041645 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.041196    4882 scope.go:122] "RemoveContainer" containerID="232d55f2aa3107cd13463fd49191c33296059150c9534cfed8c047cd9750d4b1"
Oct 30 22:48:54.195275 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.195241    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:48:54.201626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.201574    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-7a134d-694b898ffc-ftqdn"]
Oct 30 22:48:54.313353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.313266    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:48:54.319371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.319339    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-7a134d-d7bc4dc7f-s2nl4"]
Oct 30 22:48:54.943052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.943013    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" path="/var/lib/kubelet/pods/3c77ac22-68a0-46a7-b7f6-3e378e9874df/volumes"
Oct 30 22:48:54.943443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:54.943430    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" path="/var/lib/kubelet/pods/9b0aed04-7985-4bd7-8602-ad9a1fdb8644/volumes"
Oct 30 22:48:55.581948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:55.581909    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:48:57.589687 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.589641    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" podUID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" containerName="payload-processing" containerID="cri-o://908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76" gracePeriod=25
Oct 30 22:48:57.697353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.697304    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" podUID="ee18f269-aa08-4805-bbdd-10d7706f1707" containerName="payload-pre-processing" containerID="cri-o://f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b" gracePeriod=25
Oct 30 22:48:57.859666 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.859458    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:48:57.965903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.965855    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:48:57.974992 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.974960    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\") pod \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") "
Oct 30 22:48:57.975189 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.975170    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\" (UniqueName: \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\") pod \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\" (UID: \"3260293a-41f9-44a5-aaa0-1bc61d2d159f\") "
Oct 30 22:48:57.975423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.975396    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume" pod "3260293a-41f9-44a5-aaa0-1bc61d2d159f" (UID: "3260293a-41f9-44a5-aaa0-1bc61d2d159f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:57.975715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.975686    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3260293a-41f9-44a5-aaa0-1bc61d2d159f-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:57.977821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:57.977794    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl" pod "3260293a-41f9-44a5-aaa0-1bc61d2d159f" (UID: "3260293a-41f9-44a5-aaa0-1bc61d2d159f"). InnerVolumeSpecName "kube-api-access-rdswl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:58.067916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.067882    4882 generic.go:356] "Generic (PLEG): container finished" podID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" containerID="908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76" exitCode=0
Oct 30 22:48:58.068094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.067930    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"
Oct 30 22:48:58.068094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.067970    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" event={"ID":"3260293a-41f9-44a5-aaa0-1bc61d2d159f","Type":"ContainerDied","Data":"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"}
Oct 30 22:48:58.068094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.068029    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh" event={"ID":"3260293a-41f9-44a5-aaa0-1bc61d2d159f","Type":"ContainerDied","Data":"3d4408ddab6829035f6937475cc949cc3cad9fe9d9c4c72b8da4317fe940c2e4"}
Oct 30 22:48:58.068094 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.068053    4882 scope.go:122] "RemoveContainer" containerID="908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"
Oct 30 22:48:58.069675 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.069649    4882 generic.go:356] "Generic (PLEG): container finished" podID="ee18f269-aa08-4805-bbdd-10d7706f1707" containerID="f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b" exitCode=0
Oct 30 22:48:58.069791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.069693    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"
Oct 30 22:48:58.069791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.069707    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" event={"ID":"ee18f269-aa08-4805-bbdd-10d7706f1707","Type":"ContainerDied","Data":"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"}
Oct 30 22:48:58.069791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.069747    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz" event={"ID":"ee18f269-aa08-4805-bbdd-10d7706f1707","Type":"ContainerDied","Data":"b546da6cc1a1fddcdef017c858c457ba689e59e68bc395fb53db1c2e8a030300"}
Oct 30 22:48:58.076465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.076442    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\" (UniqueName: \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\") pod \"ee18f269-aa08-4805-bbdd-10d7706f1707\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") "
Oct 30 22:48:58.076630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.076568    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\") pod \"ee18f269-aa08-4805-bbdd-10d7706f1707\" (UID: \"ee18f269-aa08-4805-bbdd-10d7706f1707\") "
Oct 30 22:48:58.076964 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.076941    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume" pod "ee18f269-aa08-4805-bbdd-10d7706f1707" (UID: "ee18f269-aa08-4805-bbdd-10d7706f1707"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:48:58.077059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.077037    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee18f269-aa08-4805-bbdd-10d7706f1707-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.077122 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.077058    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rdswl\" (UniqueName: \"kubernetes.io/projected/3260293a-41f9-44a5-aaa0-1bc61d2d159f-kube-api-access-rdswl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.078936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.078911    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r" pod "ee18f269-aa08-4805-bbdd-10d7706f1707" (UID: "ee18f269-aa08-4805-bbdd-10d7706f1707"). InnerVolumeSpecName "kube-api-access-jnh8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:48:58.079196 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.079179    4882 scope.go:122] "RemoveContainer" containerID="908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"
Oct 30 22:48:58.079498 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:58.079480    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76\": container with ID starting with 908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76 not found: ID does not exist" containerID="908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"
Oct 30 22:48:58.079564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.079505    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76"} err="failed to get container status \"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76\": rpc error: code = NotFound desc = could not find container \"908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76\": container with ID starting with 908ce5b534bc0dfc34c2be53ab9a2f9e5b59826feddca8244dec0655b5c04b76 not found: ID does not exist"
Oct 30 22:48:58.079564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.079522    4882 scope.go:122] "RemoveContainer" containerID="f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"
Oct 30 22:48:58.088738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.088718    4882 scope.go:122] "RemoveContainer" containerID="f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"
Oct 30 22:48:58.089007 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:48:58.088986    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b\": container with ID starting with f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b not found: ID does not exist" containerID="f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"
Oct 30 22:48:58.089090 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.089014    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b"} err="failed to get container status \"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b\": rpc error: code = NotFound desc = could not find container \"f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b\": container with ID starting with f3f5630df804cc5d08767c53d9ff57818c95708a7dc75364abfcee5b8f65603b not found: ID does not exist"
Oct 30 22:48:58.178311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.178214    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnh8r\" (UniqueName: \"kubernetes.io/projected/ee18f269-aa08-4805-bbdd-10d7706f1707-kube-api-access-jnh8r\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:48:58.208950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.208916    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:48:58.217251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.217220    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-2cc983-6cccb7f975-96blh"]
Oct 30 22:48:58.498013 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.497921    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:48:58.502574 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.502540    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-2cc983-6f9746d7dc-5stdz"]
Oct 30 22:48:58.943720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.943684    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" path="/var/lib/kubelet/pods/3260293a-41f9-44a5-aaa0-1bc61d2d159f/volumes"
Oct 30 22:48:58.944118 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:48:58.944109    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee18f269-aa08-4805-bbdd-10d7706f1707" path="/var/lib/kubelet/pods/ee18f269-aa08-4805-bbdd-10d7706f1707/volumes"
Oct 30 22:49:00.410603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:00.410561    4882 scope.go:122] "RemoveContainer" containerID="f9f6e18abf510ac7bf90b9aa137fd15d48b7bc12082fd53419308970dd73279c"
Oct 30 22:49:00.422735 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:00.422668    4882 scope.go:122] "RemoveContainer" containerID="f5e2268be0d503240fcedef0b66a8477050ae175332ef8472b4e2c18ef61f1fb"
Oct 30 22:49:00.443791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:00.443759    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"]
Oct 30 22:49:00.494125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:00.494092    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-1a5f4a-kxzrx"]
Oct 30 22:49:00.944409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:00.944376    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06329c33-ce9b-4476-b427-9e31a346c0e2" path="/var/lib/kubelet/pods/06329c33-ce9b-4476-b427-9e31a346c0e2/volumes"
Oct 30 22:49:08.450408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450369    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"]
Oct 30 22:49:08.450939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450914    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ef76668a-6668-497f-8af0-e3d88796d54e" containerName="revoke-keys"
Oct 30 22:49:08.450939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450927    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef76668a-6668-497f-8af0-e3d88796d54e" containerName="revoke-keys"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450958    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" containerName="payload-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450964    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" containerName="payload-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450974    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" containerName="payload-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450979    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" containerName="payload-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450987    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b9b140d6-8a80-4af2-815e-da7355134862" containerName="revoke-keys"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.450991    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9b140d6-8a80-4af2-815e-da7355134862" containerName="revoke-keys"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451000    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ee18f269-aa08-4805-bbdd-10d7706f1707" containerName="payload-pre-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451005    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee18f269-aa08-4805-bbdd-10d7706f1707" containerName="payload-pre-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451013    4882 cpu_manager.go:414] "containerMap: removing container" podUID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" containerName="payload-pre-processing"
Oct 30 22:49:08.451021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451018    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" containerName="payload-pre-processing"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451116    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef76668a-6668-497f-8af0-e3d88796d54e" containerName="revoke-keys"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451127    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3260293a-41f9-44a5-aaa0-1bc61d2d159f" containerName="payload-processing"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451133    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c77ac22-68a0-46a7-b7f6-3e378e9874df" containerName="payload-processing"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451139    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9b140d6-8a80-4af2-815e-da7355134862" containerName="revoke-keys"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451144    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee18f269-aa08-4805-bbdd-10d7706f1707" containerName="payload-pre-processing"
Oct 30 22:49:08.451347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.451148    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b0aed04-7985-4bd7-8602-ad9a1fdb8644" containerName="payload-pre-processing"
Oct 30 22:49:08.454689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.454669    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.508922 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.508890    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"]
Oct 30 22:49:08.555726 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.555698    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:08.555937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.555743    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:49:08.597417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.597382    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjv8c\" (UniqueName: \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.597624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.597526    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.597624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.597566    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.698919 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.698875    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.699114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.698935    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.699114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.699071    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjv8c\" (UniqueName: \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.699764 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.699742    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.701406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.701350    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.760963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.760931    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjv8c\" (UniqueName: \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\") pod \"maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:08.768739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:08.768718    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:09.143258 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:49:09.143228    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod775c628d_4d3e_4c23_8b53_1fcf4ece9274.slice/crio-c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8 WatchSource:0}: Error finding container c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8: Status 404 returned error can't find the container with id c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8
Oct 30 22:49:09.147558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:09.147509    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"]
Oct 30 22:49:10.147745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:10.147710    4882 generic.go:356] "Generic (PLEG): container finished" podID="775c628d-4d3e-4c23-8b53-1fcf4ece9274" containerID="e67a81f0ad030700ec5c89ac4d9357e946232703d1a5a23a0b1328da8146743b" exitCode=0
Oct 30 22:49:10.148146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:10.147789    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9" event={"ID":"775c628d-4d3e-4c23-8b53-1fcf4ece9274","Type":"ContainerDied","Data":"e67a81f0ad030700ec5c89ac4d9357e946232703d1a5a23a0b1328da8146743b"}
Oct 30 22:49:10.148146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:10.147838    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9" event={"ID":"775c628d-4d3e-4c23-8b53-1fcf4ece9274","Type":"ContainerStarted","Data":"c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8"}
Oct 30 22:49:11.298326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.298298    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:11.427809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.427722    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\") pod \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") "
Oct 30 22:49:11.427973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.427882    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\") pod \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") "
Oct 30 22:49:11.428016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.427977    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\" (UniqueName: \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\") pod \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\" (UID: \"775c628d-4d3e-4c23-8b53-1fcf4ece9274\") "
Oct 30 22:49:11.428295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.428267    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca" pod "775c628d-4d3e-4c23-8b53-1fcf4ece9274" (UID: "775c628d-4d3e-4c23-8b53-1fcf4ece9274"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:11.428429 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.428389    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:11.429804 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.429776    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca" pod "775c628d-4d3e-4c23-8b53-1fcf4ece9274" (UID: "775c628d-4d3e-4c23-8b53-1fcf4ece9274"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:11.429970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.429950    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c" pod "775c628d-4d3e-4c23-8b53-1fcf4ece9274" (UID: "775c628d-4d3e-4c23-8b53-1fcf4ece9274"). InnerVolumeSpecName "kube-api-access-zjv8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:11.529396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.529359    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjv8c\" (UniqueName: \"kubernetes.io/projected/775c628d-4d3e-4c23-8b53-1fcf4ece9274-kube-api-access-zjv8c\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:11.529396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:11.529391    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/775c628d-4d3e-4c23-8b53-1fcf4ece9274-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:12.161352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:12.161319    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"
Oct 30 22:49:12.161352 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:12.161344    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9" event={"ID":"775c628d-4d3e-4c23-8b53-1fcf4ece9274","Type":"ContainerDied","Data":"c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8"}
Oct 30 22:49:12.161603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:12.161396    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c91b61042c7ef8f7add0553b488256c79842e9bb0e560fa4805e0245d91f46e8"
Oct 30 22:49:20.808738 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:49:20.808697    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:49:22.364466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:22.364430    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:49:22.471098 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:22.471061    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:49:27.364952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.364902    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" podUID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" containerName="payload-processing" containerID="cri-o://e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09" gracePeriod=25
Oct 30 22:49:27.471385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.471343    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" podUID="3ab19cf1-e870-4374-afb6-5968b253956b" containerName="payload-pre-processing" containerID="cri-o://86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1" gracePeriod=25
Oct 30 22:49:27.616464 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.616392    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:49:27.708112 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.708079    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\") pod \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") "
Oct 30 22:49:27.708273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.708205    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\" (UniqueName: \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\") pod \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\" (UID: \"a0482efe-280f-43cf-a6f7-03adcad2eb6f\") "
Oct 30 22:49:27.708471 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.708447    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume" pod "a0482efe-280f-43cf-a6f7-03adcad2eb6f" (UID: "a0482efe-280f-43cf-a6f7-03adcad2eb6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:27.710428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.710401    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9" pod "a0482efe-280f-43cf-a6f7-03adcad2eb6f" (UID: "a0482efe-280f-43cf-a6f7-03adcad2eb6f"). InnerVolumeSpecName "kube-api-access-xpxx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:27.727986 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.727961    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:49:27.809355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.809323    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\" (UniqueName: \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\") pod \"3ab19cf1-e870-4374-afb6-5968b253956b\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") "
Oct 30 22:49:27.809553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.809388    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\") pod \"3ab19cf1-e870-4374-afb6-5968b253956b\" (UID: \"3ab19cf1-e870-4374-afb6-5968b253956b\") "
Oct 30 22:49:27.809694 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.809673    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xpxx9\" (UniqueName: \"kubernetes.io/projected/a0482efe-280f-43cf-a6f7-03adcad2eb6f-kube-api-access-xpxx9\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:27.809733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.809702    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a0482efe-280f-43cf-a6f7-03adcad2eb6f-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:27.809840 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.809815    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume" pod "3ab19cf1-e870-4374-afb6-5968b253956b" (UID: "3ab19cf1-e870-4374-afb6-5968b253956b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:27.811496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.811474    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p" pod "3ab19cf1-e870-4374-afb6-5968b253956b" (UID: "3ab19cf1-e870-4374-afb6-5968b253956b"). InnerVolumeSpecName "kube-api-access-q877p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:27.911188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.911072    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q877p\" (UniqueName: \"kubernetes.io/projected/3ab19cf1-e870-4374-afb6-5968b253956b-kube-api-access-q877p\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:27.911188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:27.911124    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3ab19cf1-e870-4374-afb6-5968b253956b-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:28.266125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.266031    4882 generic.go:356] "Generic (PLEG): container finished" podID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" containerID="e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09" exitCode=0
Oct 30 22:49:28.266125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.266076    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"
Oct 30 22:49:28.266125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.266116    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" event={"ID":"a0482efe-280f-43cf-a6f7-03adcad2eb6f","Type":"ContainerDied","Data":"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"}
Oct 30 22:49:28.266432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.266171    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5" event={"ID":"a0482efe-280f-43cf-a6f7-03adcad2eb6f","Type":"ContainerDied","Data":"07a589bd9e8471690dd93ff4068987d25b12bbaa9c5142812b39aa253cc1078a"}
Oct 30 22:49:28.266432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.266197    4882 scope.go:122] "RemoveContainer" containerID="e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"
Oct 30 22:49:28.267850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.267819    4882 generic.go:356] "Generic (PLEG): container finished" podID="3ab19cf1-e870-4374-afb6-5968b253956b" containerID="86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1" exitCode=0
Oct 30 22:49:28.267988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.267864    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"
Oct 30 22:49:28.267988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.267868    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" event={"ID":"3ab19cf1-e870-4374-afb6-5968b253956b","Type":"ContainerDied","Data":"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"}
Oct 30 22:49:28.267988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.267928    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm" event={"ID":"3ab19cf1-e870-4374-afb6-5968b253956b","Type":"ContainerDied","Data":"d932daf2ad287718ab8e7956c574e5f92b3d92d019aaeb87091e30deeba9756e"}
Oct 30 22:49:28.278305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.278284    4882 scope.go:122] "RemoveContainer" containerID="e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"
Oct 30 22:49:28.278648 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:49:28.278618    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09\": container with ID starting with e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09 not found: ID does not exist" containerID="e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"
Oct 30 22:49:28.278738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.278650    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09"} err="failed to get container status \"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09\": rpc error: code = NotFound desc = could not find container \"e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09\": container with ID starting with e01a1d7024a47bcc6b86bbc8a2785f0c42e7a3a6fe555ed946b6d0cb82d9cb09 not found: ID does not exist"
Oct 30 22:49:28.278738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.278668    4882 scope.go:122] "RemoveContainer" containerID="86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"
Oct 30 22:49:28.288437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.288418    4882 scope.go:122] "RemoveContainer" containerID="86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"
Oct 30 22:49:28.288778 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:49:28.288756    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1\": container with ID starting with 86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1 not found: ID does not exist" containerID="86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"
Oct 30 22:49:28.288858 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.288787    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1"} err="failed to get container status \"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1\": rpc error: code = NotFound desc = could not find container \"86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1\": container with ID starting with 86720771e52137057d697330d502618b047660c10cd0ba04f31378cedf825da1 not found: ID does not exist"
Oct 30 22:49:28.435720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.435684    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:49:28.439460 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.439435    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-6b46c3-5b688c8bf5-vq8xm"]
Oct 30 22:49:28.553928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.553811    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:49:28.557864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.557822    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-b-6b46c3-575b687889-fw8l5"]
Oct 30 22:49:28.944658 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.944617    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ab19cf1-e870-4374-afb6-5968b253956b" path="/var/lib/kubelet/pods/3ab19cf1-e870-4374-afb6-5968b253956b/volumes"
Oct 30 22:49:28.945690 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:28.945666    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" path="/var/lib/kubelet/pods/a0482efe-280f-43cf-a6f7-03adcad2eb6f/volumes"
Oct 30 22:49:43.466865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:43.465993    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": read tcp 10.132.0.2:51566->10.132.1.83:15021: read: connection reset by peer" start-of-body=
Oct 30 22:49:43.466865 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:43.466066    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": read tcp 10.132.0.2:51566->10.132.1.83:15021: read: connection reset by peer"
Oct 30 22:49:44.393528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.393492    4882 generic.go:356] "Generic (PLEG): container finished" podID="4f04c653-84be-4ae7-94c7-99a00323d294" containerID="9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789" exitCode=0
Oct 30 22:49:44.393743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.393568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerDied","Data":"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"}
Oct 30 22:49:44.394093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.394065    4882 scope.go:122] "RemoveContainer" containerID="9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"
Oct 30 22:49:44.395667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.395639    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:49:44.395733 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.395711    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:49:44.395774 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:44.395744    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:49:45.402044 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:45.402011    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerStarted","Data":"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"}
Oct 30 22:49:45.755860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:45.755750    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:49:45.756052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:45.756018    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:45.756111 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:45.756088    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:46.756059 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:46.756015    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:46.756477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:46.756084    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:47.755999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:47.755961    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:47.756344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:47.756301    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:48.519820 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:48.519772    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"]
Oct 30 22:49:48.569902 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:48.569850    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-7a134d-qcvfc"]
Oct 30 22:49:48.758076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:48.758038    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:48.758699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:48.758662    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:48.954611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:48.954134    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8f69f93-b308-4673-889c-e3d56374e2a8" path="/var/lib/kubelet/pods/f8f69f93-b308-4673-889c-e3d56374e2a8/volumes"
Oct 30 22:49:49.755393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:49.755356    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:49.755612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:49.755422    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:49.796340 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:49.796310    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"]
Oct 30 22:49:49.850646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:49.850417    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-3c4802-wfzwn"]
Oct 30 22:49:50.755734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:50.755690    4882 patch_prober.go:29] interesting pod/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused" start-of-body=
Oct 30 22:49:50.755953 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:50.755775    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.83:15021/healthz/ready\": dial tcp 10.132.1.83:15021: connect: connection refused"
Oct 30 22:49:50.812054 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:49:50.812012    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:49:50.947128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:50.947059    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506ad38e-9913-490f-b454-8742653ba30e" path="/var/lib/kubelet/pods/506ad38e-9913-490f-b454-8742653ba30e/volumes"
Oct 30 22:49:51.608572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:51.608537    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"]
Oct 30 22:49:51.658527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:51.658484    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2cc983-qkhzb"]
Oct 30 22:49:51.759970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:51.759939    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:49:51.760389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:51.760365    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:49:51.761165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:51.761143    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:49:52.948428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:52.948388    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef76668a-6668-497f-8af0-e3d88796d54e" path="/var/lib/kubelet/pods/ef76668a-6668-497f-8af0-e3d88796d54e/volumes"
Oct 30 22:49:55.578025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:55.577979    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:49:55.578554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:55.578271    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy" containerID="cri-o://bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85" gracePeriod=30
Oct 30 22:49:56.430146 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430109    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"]
Oct 30 22:49:56.430686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430672    4882 cpu_manager.go:414] "containerMap: removing container" podUID="775c628d-4d3e-4c23-8b53-1fcf4ece9274" containerName="revoke-keys"
Oct 30 22:49:56.430744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430689    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="775c628d-4d3e-4c23-8b53-1fcf4ece9274" containerName="revoke-keys"
Oct 30 22:49:56.430744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430718    4882 cpu_manager.go:414] "containerMap: removing container" podUID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" containerName="payload-processing"
Oct 30 22:49:56.430744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430724    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" containerName="payload-processing"
Oct 30 22:49:56.430744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430735    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3ab19cf1-e870-4374-afb6-5968b253956b" containerName="payload-pre-processing"
Oct 30 22:49:56.430744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430740    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ab19cf1-e870-4374-afb6-5968b253956b" containerName="payload-pre-processing"
Oct 30 22:49:56.430909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430810    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="775c628d-4d3e-4c23-8b53-1fcf4ece9274" containerName="revoke-keys"
Oct 30 22:49:56.430909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430825    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0482efe-280f-43cf-a6f7-03adcad2eb6f" containerName="payload-processing"
Oct 30 22:49:56.430909 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.430833    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ab19cf1-e870-4374-afb6-5968b253956b" containerName="payload-pre-processing"
Oct 30 22:49:56.435477 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.435454    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.488083 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.488046    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"]
Oct 30 22:49:56.488400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.488369    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:49:56.488678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.488658    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:49:56.560770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.560727    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.560958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.560773    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7gx8c\" (UniqueName: \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.560958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.560884    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.662404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.662363    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.662864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.662414    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7gx8c\" (UniqueName: \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.662864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.662460    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.663078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.663056    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.664833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.664812    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.727646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.727556    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7gx8c\" (UniqueName: \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:56.748984 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:56.748950    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:57.123779 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:49:57.123743    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc110c24e_e2df_4584_9c93_1dbf551846ab.slice/crio-03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34 WatchSource:0}: Error finding container 03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34: Status 404 returned error can't find the container with id 03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34
Oct 30 22:49:57.126142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:57.126112    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"]
Oct 30 22:49:57.503835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:57.503798    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc" event={"ID":"c110c24e-e2df-4584-9c93-1dbf551846ab","Type":"ContainerStarted","Data":"2e826b4f5dccd5f737907eae107d9fcdd5e0165a4ea3761638bdbdaec3e53ba0"}
Oct 30 22:49:57.504024 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:57.503856    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc" event={"ID":"c110c24e-e2df-4584-9c93-1dbf551846ab","Type":"ContainerStarted","Data":"03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34"}
Oct 30 22:49:57.621578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:57.621520    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc" podStartSLOduration=1.621505552 podStartE2EDuration="1.621505552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:49:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:49:57.620160573 +0000 UTC m=+4320.793485000" watchObservedRunningTime="2026-10-30 22:49:57.621505552 +0000 UTC m=+4320.794829976"
Oct 30 22:49:58.513019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:58.512979    4882 generic.go:356] "Generic (PLEG): container finished" podID="c110c24e-e2df-4584-9c93-1dbf551846ab" containerID="2e826b4f5dccd5f737907eae107d9fcdd5e0165a4ea3761638bdbdaec3e53ba0" exitCode=0
Oct 30 22:49:58.513423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:58.513065    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc" event={"ID":"c110c24e-e2df-4584-9c93-1dbf551846ab","Type":"ContainerDied","Data":"2e826b4f5dccd5f737907eae107d9fcdd5e0165a4ea3761638bdbdaec3e53ba0"}
Oct 30 22:49:59.666962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.666935    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:49:59.797654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.797537    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\") pod \"c110c24e-e2df-4584-9c93-1dbf551846ab\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") "
Oct 30 22:49:59.797654 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.797617    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\" (UniqueName: \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\") pod \"c110c24e-e2df-4584-9c93-1dbf551846ab\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") "
Oct 30 22:49:59.797928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.797749    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\") pod \"c110c24e-e2df-4584-9c93-1dbf551846ab\" (UID: \"c110c24e-e2df-4584-9c93-1dbf551846ab\") "
Oct 30 22:49:59.798168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.798132    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca" pod "c110c24e-e2df-4584-9c93-1dbf551846ab" (UID: "c110c24e-e2df-4584-9c93-1dbf551846ab"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:49:59.799918 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.799889    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca" pod "c110c24e-e2df-4584-9c93-1dbf551846ab" (UID: "c110c24e-e2df-4584-9c93-1dbf551846ab"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:49:59.800043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.800017    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c" pod "c110c24e-e2df-4584-9c93-1dbf551846ab" (UID: "c110c24e-e2df-4584-9c93-1dbf551846ab"). InnerVolumeSpecName "kube-api-access-7gx8c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:49:59.899244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.899209    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:59.899244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.899239    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7gx8c\" (UniqueName: \"kubernetes.io/projected/c110c24e-e2df-4584-9c93-1dbf551846ab-kube-api-access-7gx8c\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:49:59.899244 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:49:59.899250    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c110c24e-e2df-4584-9c93-1dbf551846ab-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.129765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.129730    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"]
Oct 30 22:50:00.130347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.130332    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c110c24e-e2df-4584-9c93-1dbf551846ab" containerName="revoke-keys"
Oct 30 22:50:00.130404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.130348    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c110c24e-e2df-4584-9c93-1dbf551846ab" containerName="revoke-keys"
Oct 30 22:50:00.130447 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.130427    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c110c24e-e2df-4584-9c93-1dbf551846ab" containerName="revoke-keys"
Oct 30 22:50:00.133937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.133908    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.188669 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.188633    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"]
Oct 30 22:50:00.202074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.202033    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.202259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.202093    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.202259 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.202230    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8lxn\" (UniqueName: \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.303542 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.303494    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.303793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.303553    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.303793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.303623    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8lxn\" (UniqueName: \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.304257 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.304233    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.305955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.305932    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.365908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.365857    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8lxn\" (UniqueName: \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\") pod \"maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.447989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.447900    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:00.527414 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.527387    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"
Oct 30 22:50:00.527609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.527409    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc" event={"ID":"c110c24e-e2df-4584-9c93-1dbf551846ab","Type":"ContainerDied","Data":"03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34"}
Oct 30 22:50:00.527609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.527455    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="03ba0acc1ec2d34b38d89cb7155e8adfd6a39f500beb955488de56c895f92e34"
Oct 30 22:50:00.826156 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:50:00.826124    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2dbb3d_0097_415b_b2a9_348412f9b082.slice/crio-6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b WatchSource:0}: Error finding container 6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b: Status 404 returned error can't find the container with id 6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b
Oct 30 22:50:00.828943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.828910    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"]
Oct 30 22:50:00.846982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.846959    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:50:00.909646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909616    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.909818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909659    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.909818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909714    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.909818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909736    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.909818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909787    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.909818 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909808    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.910116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909835    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.910116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909876    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.910116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909920    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\") pod \"54398249-098e-432e-b1bf-2ae76b58d0ad\" (UID: \"54398249-098e-432e-b1bf-2ae76b58d0ad\") "
Oct 30 22:50:00.910116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.909956    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:00.910370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.910348    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:00.910885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.910668    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:00.911668 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.911124    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:00.911917 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.911857    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:00.912229 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.912199    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:00.912664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.912608    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:00.913088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.913059    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:00.913346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.913326    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "kube-api-access-trzsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:00.913864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:00.913825    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo" pod "54398249-098e-432e-b1bf-2ae76b58d0ad" (UID: "54398249-098e-432e-b1bf-2ae76b58d0ad"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:50:01.011484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011446    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011475    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011484    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011495    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011504    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54398249-098e-432e-b1bf-2ae76b58d0ad-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011516    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011527    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54398249-098e-432e-b1bf-2ae76b58d0ad-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.011773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.011535    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-trzsk\" (UniqueName: \"kubernetes.io/projected/54398249-098e-432e-b1bf-2ae76b58d0ad-kube-api-access-trzsk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:01.535270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.535232    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx" event={"ID":"ca2dbb3d-0097-415b-b2a9-348412f9b082","Type":"ContainerStarted","Data":"7ed5ef07bd58fd63b37c8fbebab9732c684970cb46c1367166a75c2dfe8750eb"}
Oct 30 22:50:01.535479 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.535298    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx" event={"ID":"ca2dbb3d-0097-415b-b2a9-348412f9b082","Type":"ContainerStarted","Data":"6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b"}
Oct 30 22:50:01.537042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.537017    4882 generic.go:356] "Generic (PLEG): container finished" podID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerID="bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85" exitCode=0
Oct 30 22:50:01.537179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.537061    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"
Oct 30 22:50:01.537179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.537093    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerDied","Data":"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"}
Oct 30 22:50:01.537179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.537138    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m" event={"ID":"54398249-098e-432e-b1bf-2ae76b58d0ad","Type":"ContainerDied","Data":"d21f8916e83212073876c1613bb1546f5c046fd053e58af2fdd7ee2fb9976edc"}
Oct 30 22:50:01.537179 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.537159    4882 scope.go:122] "RemoveContainer" containerID="bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"
Oct 30 22:50:01.548159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.548140    4882 scope.go:122] "RemoveContainer" containerID="57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"
Oct 30 22:50:01.557908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.557867    4882 scope.go:122] "RemoveContainer" containerID="bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"
Oct 30 22:50:01.558195 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:01.558172    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85\": container with ID starting with bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85 not found: ID does not exist" containerID="bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"
Oct 30 22:50:01.558248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.558204    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85"} err="failed to get container status \"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85\": rpc error: code = NotFound desc = could not find container \"bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85\": container with ID starting with bf03517e7c29ff08faa781645b581b81d78c957b656b9525c11d9b66703e4c85 not found: ID does not exist"
Oct 30 22:50:01.558248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.558224    4882 scope.go:122] "RemoveContainer" containerID="57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"
Oct 30 22:50:01.558449 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:01.558434    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea\": container with ID starting with 57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea not found: ID does not exist" containerID="57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"
Oct 30 22:50:01.558491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.558455    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea"} err="failed to get container status \"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea\": rpc error: code = NotFound desc = could not find container \"57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea\": container with ID starting with 57fcb602a3708bee6b9079981e0ce93a3601d13834c7700056d5b440fe7b31ea not found: ID does not exist"
Oct 30 22:50:01.653753 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.653683    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx" podStartSLOduration=1.653666684 podStartE2EDuration="1.653666684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:01.652370524 +0000 UTC m=+4324.825694949" watchObservedRunningTime="2026-10-30 22:50:01.653666684 +0000 UTC m=+4324.826991108"
Oct 30 22:50:01.756698 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:01.756665    4882 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca2dbb3d_0097_415b_b2a9_348412f9b082.slice/crio-7ed5ef07bd58fd63b37c8fbebab9732c684970cb46c1367166a75c2dfe8750eb.scope\": RecentStats: unable to find data in memory cache]"
Oct 30 22:50:01.767782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.767749    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:50:01.773629 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:01.773580    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-b-7a134d-openshift-default-559bcb57b5-wt56m"]
Oct 30 22:50:02.544527 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:02.544492    4882 generic.go:356] "Generic (PLEG): container finished" podID="ca2dbb3d-0097-415b-b2a9-348412f9b082" containerID="7ed5ef07bd58fd63b37c8fbebab9732c684970cb46c1367166a75c2dfe8750eb" exitCode=0
Oct 30 22:50:02.544994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:02.544579    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx" event={"ID":"ca2dbb3d-0097-415b-b2a9-348412f9b082","Type":"ContainerDied","Data":"7ed5ef07bd58fd63b37c8fbebab9732c684970cb46c1367166a75c2dfe8750eb"}
Oct 30 22:50:02.942896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:02.942858    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" path="/var/lib/kubelet/pods/54398249-098e-432e-b1bf-2ae76b58d0ad/volumes"
Oct 30 22:50:03.696618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.696574    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:03.841418 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.841388    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\") pod \"ca2dbb3d-0097-415b-b2a9-348412f9b082\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") "
Oct 30 22:50:03.841582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.841429    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\") pod \"ca2dbb3d-0097-415b-b2a9-348412f9b082\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") "
Oct 30 22:50:03.841678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.841649    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\" (UniqueName: \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\") pod \"ca2dbb3d-0097-415b-b2a9-348412f9b082\" (UID: \"ca2dbb3d-0097-415b-b2a9-348412f9b082\") "
Oct 30 22:50:03.841830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.841808    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca" pod "ca2dbb3d-0097-415b-b2a9-348412f9b082" (UID: "ca2dbb3d-0097-415b-b2a9-348412f9b082"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:03.842086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.842067    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.843736 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.843704    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca" pod "ca2dbb3d-0097-415b-b2a9-348412f9b082" (UID: "ca2dbb3d-0097-415b-b2a9-348412f9b082"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:03.843845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.843764    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn" pod "ca2dbb3d-0097-415b-b2a9-348412f9b082" (UID: "ca2dbb3d-0097-415b-b2a9-348412f9b082"). InnerVolumeSpecName "kube-api-access-g8lxn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:03.943144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.943105    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8lxn\" (UniqueName: \"kubernetes.io/projected/ca2dbb3d-0097-415b-b2a9-348412f9b082-kube-api-access-g8lxn\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:03.943144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:03.943137    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ca2dbb3d-0097-415b-b2a9-348412f9b082-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:04.560358 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:04.560321    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"
Oct 30 22:50:04.560543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:04.560357    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx" event={"ID":"ca2dbb3d-0097-415b-b2a9-348412f9b082","Type":"ContainerDied","Data":"6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b"}
Oct 30 22:50:04.560543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:04.560410    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d161760206d51fd0539a9c72e485e24140e827eac4c6af0bc280a754ebfdd5b"
Oct 30 22:50:04.597263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:04.597222    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 22:50:07.891756 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:07.891675    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:50:08.049067 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:08.049031    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:50:11.025484 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.025454    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"]
Oct 30 22:50:11.026088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026069    4882 cpu_manager.go:414] "containerMap: removing container" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.026088 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026086    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026099    4882 cpu_manager.go:414] "containerMap: removing container" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026105    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026126    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ca2dbb3d-0097-415b-b2a9-348412f9b082" containerName="revoke-keys"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026130    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca2dbb3d-0097-415b-b2a9-348412f9b082" containerName="revoke-keys"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026229    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca2dbb3d-0097-415b-b2a9-348412f9b082" containerName="revoke-keys"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026245    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.026260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.026251    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="54398249-098e-432e-b1bf-2ae76b58d0ad" containerName="istio-proxy"
Oct 30 22:50:11.029681 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.029660    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"]
Oct 30 22:50:11.029823 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.029788    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.033761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.033738    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.082931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.082892    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:11.083110 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.083028    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:50:11.122300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122256    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.122501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122333    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.122501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122394    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24nd4\" (UniqueName: \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.122501 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122438    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.122770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122543    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.122770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.122573    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzgbh\" (UniqueName: \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.123139 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.123117    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"]
Oct 30 22:50:11.136445 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.136413    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"]
Oct 30 22:50:11.224057 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224019    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.224344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224113    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.224344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224155    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzgbh\" (UniqueName: \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.224344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224242    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.224344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224294    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.224344 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224335    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24nd4\" (UniqueName: \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.224821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224795    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.224960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.224869    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.226713 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.226686    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.226844 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.226744    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.285950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.285850    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-24nd4\" (UniqueName: \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\") pod \"maas-api-revoke-sub-10fef9d9-pnxfq\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.286128 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.285947    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzgbh\" (UniqueName: \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\") pod \"maas-api-revoke-sub-8fa93289-sgjvf\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.347977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.347928    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:11.354331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.354308    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:11.730504 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:50:11.730471    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52296f8f_9157_49df_aa82_b56f2f04e8a1.slice/crio-699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48 WatchSource:0}: Error finding container 699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48: Status 404 returned error can't find the container with id 699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48
Oct 30 22:50:11.733328 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.733282    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"]
Oct 30 22:50:11.767417 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.767379    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"]
Oct 30 22:50:11.770833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.770806    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-6b46c3-29rh9"]
Oct 30 22:50:11.775973 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:11.775944    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"]
Oct 30 22:50:12.614243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.614209    4882 generic.go:356] "Generic (PLEG): container finished" podID="52296f8f-9157-49df-aa82-b56f2f04e8a1" containerID="43ba58a5c99abf402874aea3c0e728f51ea64b4dfce436c3e70a734d696767aa" exitCode=0
Oct 30 22:50:12.614723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.614293    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf" event={"ID":"52296f8f-9157-49df-aa82-b56f2f04e8a1","Type":"ContainerDied","Data":"43ba58a5c99abf402874aea3c0e728f51ea64b4dfce436c3e70a734d696767aa"}
Oct 30 22:50:12.614723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.614362    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf" event={"ID":"52296f8f-9157-49df-aa82-b56f2f04e8a1","Type":"ContainerStarted","Data":"699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48"}
Oct 30 22:50:12.615958 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.615930    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq" event={"ID":"ed491cc6-f903-439f-80a0-4bf0e3e38ca1","Type":"ContainerStarted","Data":"03f1e00454a818bd68f7be606e97ae7b6ae23624be8accb6573b7f2f1160e784"}
Oct 30 22:50:12.616072 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.615979    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq" event={"ID":"ed491cc6-f903-439f-80a0-4bf0e3e38ca1","Type":"ContainerStarted","Data":"a05c85d9963dfa2780257a5359c2416563284d7628e2fc1452e8c959b538c332"}
Oct 30 22:50:12.863897 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.863784    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq" podStartSLOduration=2.863766585 podStartE2EDuration="2.863766585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:12.862937148 +0000 UTC m=+4336.036261574" watchObservedRunningTime="2026-10-30 22:50:12.863766585 +0000 UTC m=+4336.037091013"
Oct 30 22:50:12.943095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:12.943061    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="775c628d-4d3e-4c23-8b53-1fcf4ece9274" path="/var/lib/kubelet/pods/775c628d-4d3e-4c23-8b53-1fcf4ece9274/volumes"
Oct 30 22:50:13.050329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.050290    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing" containerID="cri-o://82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6" gracePeriod=25
Oct 30 22:50:13.624526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.624487    4882 generic.go:356] "Generic (PLEG): container finished" podID="ed491cc6-f903-439f-80a0-4bf0e3e38ca1" containerID="03f1e00454a818bd68f7be606e97ae7b6ae23624be8accb6573b7f2f1160e784" exitCode=0
Oct 30 22:50:13.624963 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.624562    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq" event={"ID":"ed491cc6-f903-439f-80a0-4bf0e3e38ca1","Type":"ContainerDied","Data":"03f1e00454a818bd68f7be606e97ae7b6ae23624be8accb6573b7f2f1160e784"}
Oct 30 22:50:13.795495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.795464    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:13.957341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.957257    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\") pod \"52296f8f-9157-49df-aa82-b56f2f04e8a1\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") "
Oct 30 22:50:13.957507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.957352    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\" (UniqueName: \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\") pod \"52296f8f-9157-49df-aa82-b56f2f04e8a1\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") "
Oct 30 22:50:13.957507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.957387    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\") pod \"52296f8f-9157-49df-aa82-b56f2f04e8a1\" (UID: \"52296f8f-9157-49df-aa82-b56f2f04e8a1\") "
Oct 30 22:50:13.957643 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.957616    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca" pod "52296f8f-9157-49df-aa82-b56f2f04e8a1" (UID: "52296f8f-9157-49df-aa82-b56f2f04e8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:13.957822 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.957802    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:13.959528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.959495    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca" pod "52296f8f-9157-49df-aa82-b56f2f04e8a1" (UID: "52296f8f-9157-49df-aa82-b56f2f04e8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:13.959692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:13.959671    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh" pod "52296f8f-9157-49df-aa82-b56f2f04e8a1" (UID: "52296f8f-9157-49df-aa82-b56f2f04e8a1"). InnerVolumeSpecName "kube-api-access-qzgbh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:14.059419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.059374    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzgbh\" (UniqueName: \"kubernetes.io/projected/52296f8f-9157-49df-aa82-b56f2f04e8a1-kube-api-access-qzgbh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.059419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.059411    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/52296f8f-9157-49df-aa82-b56f2f04e8a1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.632052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.632023    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"
Oct 30 22:50:14.632461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.632059    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf" event={"ID":"52296f8f-9157-49df-aa82-b56f2f04e8a1","Type":"ContainerDied","Data":"699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48"}
Oct 30 22:50:14.632461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.632105    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="699ae58097c0bea314920f391129a8bfdd82df25dccb923571cbce506758fb48"
Oct 30 22:50:14.770649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.770581    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:14.868115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.868078    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\") pod \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") "
Oct 30 22:50:14.868294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.868217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\" (UniqueName: \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\") pod \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") "
Oct 30 22:50:14.868294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.868271    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\") pod \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\" (UID: \"ed491cc6-f903-439f-80a0-4bf0e3e38ca1\") "
Oct 30 22:50:14.868680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.868659    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca" pod "ed491cc6-f903-439f-80a0-4bf0e3e38ca1" (UID: "ed491cc6-f903-439f-80a0-4bf0e3e38ca1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:14.870337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.870314    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca" pod "ed491cc6-f903-439f-80a0-4bf0e3e38ca1" (UID: "ed491cc6-f903-439f-80a0-4bf0e3e38ca1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:14.870407 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.870387    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4" pod "ed491cc6-f903-439f-80a0-4bf0e3e38ca1" (UID: "ed491cc6-f903-439f-80a0-4bf0e3e38ca1"). InnerVolumeSpecName "kube-api-access-24nd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:14.970489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.970454    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24nd4\" (UniqueName: \"kubernetes.io/projected/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-kube-api-access-24nd4\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.970489 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.970491    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:14.970716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:14.970511    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ed491cc6-f903-439f-80a0-4bf0e3e38ca1-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:15.639491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:15.639458    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"
Oct 30 22:50:15.639491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:15.639471    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq" event={"ID":"ed491cc6-f903-439f-80a0-4bf0e3e38ca1","Type":"ContainerDied","Data":"a05c85d9963dfa2780257a5359c2416563284d7628e2fc1452e8c959b538c332"}
Oct 30 22:50:15.640092 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:15.639523    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a05c85d9963dfa2780257a5359c2416563284d7628e2fc1452e8c959b538c332"
Oct 30 22:50:17.637575 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:17.637536    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:50:19.532388 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.532350    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"]
Oct 30 22:50:19.532912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.532896    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ed491cc6-f903-439f-80a0-4bf0e3e38ca1" containerName="revoke-keys"
Oct 30 22:50:19.532960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.532913    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed491cc6-f903-439f-80a0-4bf0e3e38ca1" containerName="revoke-keys"
Oct 30 22:50:19.532960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.532939    4882 cpu_manager.go:414] "containerMap: removing container" podUID="52296f8f-9157-49df-aa82-b56f2f04e8a1" containerName="revoke-keys"
Oct 30 22:50:19.532960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.532944    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52296f8f-9157-49df-aa82-b56f2f04e8a1" containerName="revoke-keys"
Oct 30 22:50:19.533061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.533025    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed491cc6-f903-439f-80a0-4bf0e3e38ca1" containerName="revoke-keys"
Oct 30 22:50:19.533061 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.533039    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="52296f8f-9157-49df-aa82-b56f2f04e8a1" containerName="revoke-keys"
Oct 30 22:50:19.536441 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.536422    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.590240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.590205    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"]
Oct 30 22:50:19.618180 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.618129    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main" containerID="cri-o://9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9" gracePeriod=45
Oct 30 22:50:19.637047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.637023    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:50:19.637243 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.637224    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:50:19.718141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.718107    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.718329 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.718246    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.718426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.718338    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vlfc\" (UniqueName: \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.819618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.819208    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.819618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.819299    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.819618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.819343    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vlfc\" (UniqueName: \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.820214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.820183    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.822052 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.822024    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.882035 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.881997    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vlfc\" (UniqueName: \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:19.907454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:19.907423    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 22:50:20.021565 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021529    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.021772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021660    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.021772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021708    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.021772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021747    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.021976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021833    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.021976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021860    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:20.021976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021880    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\" (UniqueName: \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\") pod \"08ee8652-249f-4705-b037-6be8f460e4ab\" (UID: \"08ee8652-249f-4705-b037-6be8f460e4ab\") "
Oct 30 22:50:20.022137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.021971    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:20.022137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.022078    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:20.022421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.022368    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.022421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.022391    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.022421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.022400    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.024754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.024712    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "kube-api-access-xkjn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:20.024885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.024783    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:20.024885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.024812    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm" pod "08ee8652-249f-4705-b037-6be8f460e4ab" (UID: "08ee8652-249f-4705-b037-6be8f460e4ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:50:20.123458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.123426    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/08ee8652-249f-4705-b037-6be8f460e4ab-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.123458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.123456    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xkjn7\" (UniqueName: \"kubernetes.io/projected/08ee8652-249f-4705-b037-6be8f460e4ab-kube-api-access-xkjn7\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.123458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.123469    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/08ee8652-249f-4705-b037-6be8f460e4ab-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:20.150107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.150074    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:20.526906 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:50:20.526875    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef68f459_d568_42fe_9f02_740b2f9a43eb.slice/crio-9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4 WatchSource:0}: Error finding container 9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4: Status 404 returned error can't find the container with id 9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4
Oct 30 22:50:20.529369 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.529337    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"]
Oct 30 22:50:20.672006 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.671972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk" event={"ID":"ef68f459-d568-42fe-9f02-740b2f9a43eb","Type":"ContainerStarted","Data":"9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4"}
Oct 30 22:50:20.673466 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.673438    4882 generic.go:356] "Generic (PLEG): container finished" podID="08ee8652-249f-4705-b037-6be8f460e4ab" containerID="9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9" exitCode=0
Oct 30 22:50:20.673603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.673481    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" event={"ID":"08ee8652-249f-4705-b037-6be8f460e4ab","Type":"ContainerDied","Data":"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"}
Oct 30 22:50:20.673603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.673486    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"
Oct 30 22:50:20.673603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.673506    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc" event={"ID":"08ee8652-249f-4705-b037-6be8f460e4ab","Type":"ContainerDied","Data":"149e88a0f31e2e042f5720cfcc3c3ae6ef1de01ce5e66ec96f724010ddfca023"}
Oct 30 22:50:20.673603 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.673527    4882 scope.go:122] "RemoveContainer" containerID="9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"
Oct 30 22:50:20.683869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.683845    4882 scope.go:122] "RemoveContainer" containerID="9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"
Oct 30 22:50:20.684173 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:20.684154    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9\": container with ID starting with 9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9 not found: ID does not exist" containerID="9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"
Oct 30 22:50:20.684222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.684182    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9"} err="failed to get container status \"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9\": rpc error: code = NotFound desc = could not find container \"9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9\": container with ID starting with 9ab407dc7fe7553ce9774b5a2063a2ba990627496c85bb056be57bfae7a484c9 not found: ID does not exist"
Oct 30 22:50:20.809716 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:20.809611    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:50:20.853023 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.852996    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 22:50:20.903568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.903531    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-26743a/simulator-w4-26743a-kserve-69759cd7c8-fk4qc"]
Oct 30 22:50:20.951077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:20.945358    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" path="/var/lib/kubelet/pods/08ee8652-249f-4705-b037-6be8f460e4ab/volumes"
Oct 30 22:50:21.681989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:21.681887    4882 generic.go:356] "Generic (PLEG): container finished" podID="ef68f459-d568-42fe-9f02-740b2f9a43eb" containerID="796dff028a5f6b5fcd4d908492ead515be86665d080a9b431eeb7e85d737fea0" exitCode=0
Oct 30 22:50:21.681989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:21.681972    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk" event={"ID":"ef68f459-d568-42fe-9f02-740b2f9a43eb","Type":"ContainerDied","Data":"796dff028a5f6b5fcd4d908492ead515be86665d080a9b431eeb7e85d737fea0"}
Oct 30 22:50:22.834400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.834376    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:22.852691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.852660    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\") pod \"ef68f459-d568-42fe-9f02-740b2f9a43eb\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") "
Oct 30 22:50:22.852868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.852708    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\") pod \"ef68f459-d568-42fe-9f02-740b2f9a43eb\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") "
Oct 30 22:50:22.852868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.852741    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\" (UniqueName: \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\") pod \"ef68f459-d568-42fe-9f02-740b2f9a43eb\" (UID: \"ef68f459-d568-42fe-9f02-740b2f9a43eb\") "
Oct 30 22:50:22.853105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.853078    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca" pod "ef68f459-d568-42fe-9f02-740b2f9a43eb" (UID: "ef68f459-d568-42fe-9f02-740b2f9a43eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:22.853266 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.853247    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:22.854945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.854914    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca" pod "ef68f459-d568-42fe-9f02-740b2f9a43eb" (UID: "ef68f459-d568-42fe-9f02-740b2f9a43eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:50:22.854945 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.854920    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc" pod "ef68f459-d568-42fe-9f02-740b2f9a43eb" (UID: "ef68f459-d568-42fe-9f02-740b2f9a43eb"). InnerVolumeSpecName "kube-api-access-6vlfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:22.954702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.954676    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ef68f459-d568-42fe-9f02-740b2f9a43eb-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:22.954702 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:22.954702    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vlfc\" (UniqueName: \"kubernetes.io/projected/ef68f459-d568-42fe-9f02-740b2f9a43eb-kube-api-access-6vlfc\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:23.701976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:23.701943    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"
Oct 30 22:50:23.702209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:23.701962    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk" event={"ID":"ef68f459-d568-42fe-9f02-740b2f9a43eb","Type":"ContainerDied","Data":"9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4"}
Oct 30 22:50:23.702209 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:23.702036    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9814d416ceeef590a4c110aae16a45ab002147566c1d500078e1d180c881f6d4"
Oct 30 22:50:27.637093 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:27.637054    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:50:31.531435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.531391    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:50:31.532188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532165    4882 cpu_manager.go:414] "containerMap: removing container" podUID="ef68f459-d568-42fe-9f02-740b2f9a43eb" containerName="revoke-keys"
Oct 30 22:50:31.532300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532190    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef68f459-d568-42fe-9f02-740b2f9a43eb" containerName="revoke-keys"
Oct 30 22:50:31.532300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532224    4882 cpu_manager.go:414] "containerMap: removing container" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main"
Oct 30 22:50:31.532300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532232    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main"
Oct 30 22:50:31.532530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532367    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef68f459-d568-42fe-9f02-740b2f9a43eb" containerName="revoke-keys"
Oct 30 22:50:31.532530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.532387    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="08ee8652-249f-4705-b037-6be8f460e4ab" containerName="main"
Oct 30 22:50:31.536208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.536183    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.591201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.591154    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:50:31.637921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.637882    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-extmod-b-9af9e7-dockercfg-r7wcj\""
Oct 30 22:50:31.637921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.637882    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-extmod-b-9af9e7\""
Oct 30 22:50:31.650743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.650705    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdt8w\" (UniqueName: \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.650949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.650925    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.751686 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.751650    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdt8w\" (UniqueName: \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.751885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.751806    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.752528 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.752504    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.813830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.813754    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdt8w\" (UniqueName: \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\") pod \"payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") " pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:31.850053 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:31.850013    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:32.228812 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:50:32.226518    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e254661_d49d_4325_96f5_cde04db4358b.slice/crio-4fe42fae172f6a526d80c700420a719d163295858a5b5a141544ae13d868ee82 WatchSource:0}: Error finding container 4fe42fae172f6a526d80c700420a719d163295858a5b5a141544ae13d868ee82: Status 404 returned error can't find the container with id 4fe42fae172f6a526d80c700420a719d163295858a5b5a141544ae13d868ee82
Oct 30 22:50:32.228812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:32.228750    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:50:32.763999 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:32.763947    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" event={"ID":"5e254661-d49d-4325-96f5-cde04db4358b","Type":"ContainerStarted","Data":"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"}
Oct 30 22:50:32.764421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:32.764172    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:32.764421 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:32.764214    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" event={"ID":"5e254661-d49d-4325-96f5-cde04db4358b","Type":"ContainerStarted","Data":"4fe42fae172f6a526d80c700420a719d163295858a5b5a141544ae13d868ee82"}
Oct 30 22:50:32.883300 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:32.883241    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" podStartSLOduration=1.883227497 podStartE2EDuration="1.883227497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:50:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:50:32.881288068 +0000 UTC m=+4356.054612497" watchObservedRunningTime="2026-10-30 22:50:32.883227497 +0000 UTC m=+4356.056551923"
Oct 30 22:50:33.317385 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.317318    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:50:33.367040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.366996    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\") pod \"8783e61f-16a1-404a-a299-0abd5e67cfce\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") "
Oct 30 22:50:33.367207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.367078    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\" (UniqueName: \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\") pod \"8783e61f-16a1-404a-a299-0abd5e67cfce\" (UID: \"8783e61f-16a1-404a-a299-0abd5e67cfce\") "
Oct 30 22:50:33.367442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.367417    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume" pod "8783e61f-16a1-404a-a299-0abd5e67cfce" (UID: "8783e61f-16a1-404a-a299-0abd5e67cfce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:50:33.369283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.369257    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf" pod "8783e61f-16a1-404a-a299-0abd5e67cfce" (UID: "8783e61f-16a1-404a-a299-0abd5e67cfce"). InnerVolumeSpecName "kube-api-access-r8jxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:50:33.468157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.468121    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8783e61f-16a1-404a-a299-0abd5e67cfce-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:33.468157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.468154    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8jxf\" (UniqueName: \"kubernetes.io/projected/8783e61f-16a1-404a-a299-0abd5e67cfce-kube-api-access-r8jxf\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:50:33.772444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.772398    4882 generic.go:356] "Generic (PLEG): container finished" podID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerID="82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6" exitCode=0
Oct 30 22:50:33.772948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.772449    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"
Oct 30 22:50:33.772948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.772472    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" event={"ID":"8783e61f-16a1-404a-a299-0abd5e67cfce","Type":"ContainerDied","Data":"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"}
Oct 30 22:50:33.772948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.772534    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6" event={"ID":"8783e61f-16a1-404a-a299-0abd5e67cfce","Type":"ContainerDied","Data":"7c366ec116d40e95b494761718713da216e6c7b27870d4d59b05d08b54ef2af5"}
Oct 30 22:50:33.772948 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.772563    4882 scope.go:122] "RemoveContainer" containerID="82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"
Oct 30 22:50:33.783886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.783863    4882 scope.go:122] "RemoveContainer" containerID="82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"
Oct 30 22:50:33.784213 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:33.784194    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6\": container with ID starting with 82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6 not found: ID does not exist" containerID="82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"
Oct 30 22:50:33.784270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.784222    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6"} err="failed to get container status \"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6\": rpc error: code = NotFound desc = could not find container \"82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6\": container with ID starting with 82d7d432655b3a7124eee34b978a918bf55b96746dce47afe6e8a7fc8f16c3f6 not found: ID does not exist"
Oct 30 22:50:33.940305 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.940272    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:50:33.990797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:33.990764    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-5e884c-74b45dd67f-z9qf6"]
Oct 30 22:50:34.944216 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:34.944179    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" path="/var/lib/kubelet/pods/8783e61f-16a1-404a-a299-0abd5e67cfce/volumes"
Oct 30 22:50:43.776497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:43.776461    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:50:50.808923 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:50:50.808880    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:50:56.457785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:56.457750    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"]
Oct 30 22:50:56.507237 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:56.507199    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-5e884c-qzzcc"]
Oct 30 22:50:56.944296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:50:56.944262    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c110c24e-e2df-4584-9c93-1dbf551846ab" path="/var/lib/kubelet/pods/c110c24e-e2df-4584-9c93-1dbf551846ab/volumes"
Oct 30 22:51:02.946699 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:02.946665    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"]
Oct 30 22:51:02.997164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:02.997127    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-26743a-nmpmx"]
Oct 30 22:51:04.278739 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.278699    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:04.279260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.279247    4882 cpu_manager.go:414] "containerMap: removing container" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing"
Oct 30 22:51:04.279307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.279261    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing"
Oct 30 22:51:04.279386 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.279377    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="8783e61f-16a1-404a-a299-0abd5e67cfce" containerName="payload-pre-processing"
Oct 30 22:51:04.284367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.284341    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.308914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.308865    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.308932    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309087    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309143    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309311 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309252    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmd2n\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309370 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309349    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309404    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309434    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.309518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.309500    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.337511 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.337466    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-26743a-openshift-default-dockercfg-lddtx\""
Oct 30 22:51:04.390913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.390865    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:04.410472 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410431    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410486    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410527    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410560    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmd2n\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410680    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410725    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410747    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.410835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410824    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.411271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.410860    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.411271 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.411252    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.411395 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.411362    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.411867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.411842    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.412004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.411849    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.412004 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.411914    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.413322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.413299    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.413423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.413408    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.473192 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.473149    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.473347 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.473224    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmd2n\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\") pod \"e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") " pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.599335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.599284    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:04.788708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.788668    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"]
Oct 30 22:51:04.794204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.794179    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.814551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.814517    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.814744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.814616    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.814744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.814656    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jk8\" (UniqueName: \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.846039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.846000    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"]
Oct 30 22:51:04.847432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.847387    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:04.847605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.847429    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:51:04.915274 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.915168    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.915475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.915289    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.915475 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.915341    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jk8\" (UniqueName: \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.916050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.916017    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.917869 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.917846    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:04.943553 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.943513    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca2dbb3d-0097-415b-b2a9-348412f9b082" path="/var/lib/kubelet/pods/ca2dbb3d-0097-415b-b2a9-348412f9b082/volumes"
Oct 30 22:51:04.977793 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:04.977758    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jk8\" (UniqueName: \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:05.068513 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:05.068481    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c07dbf8_219e_4107_b1c4_47a53bed6c7e.slice/crio-eeecc7d0839f246c0f4c3ca52992e7ebfc6181486db4b58fabe2137a97466cf2 WatchSource:0}: Error finding container eeecc7d0839f246c0f4c3ca52992e7ebfc6181486db4b58fabe2137a97466cf2: Status 404 returned error can't find the container with id eeecc7d0839f246c0f4c3ca52992e7ebfc6181486db4b58fabe2137a97466cf2
Oct 30 22:51:05.071513 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.071472    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:51:05.071620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.071565    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:51:05.071620 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.071576    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:05.071703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.071625    4882 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236060Ki","pods":"250"}
Oct 30 22:51:05.109463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.109429    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:05.488555 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:05.488516    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1b6ceb6_68aa_4801_811e_8c24402c90a7.slice/crio-98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f WatchSource:0}: Error finding container 98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f: Status 404 returned error can't find the container with id 98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f
Oct 30 22:51:05.491389 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.491365    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"]
Oct 30 22:51:05.804516 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.804471    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:51:05.989500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.989469    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8" event={"ID":"c1b6ceb6-68aa-4801-811e-8c24402c90a7","Type":"ContainerStarted","Data":"d9b62645d124d88d6c04331214e3cbb600b0eef003c9be15550661add739f832"}
Oct 30 22:51:05.989717 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.989530    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8" event={"ID":"c1b6ceb6-68aa-4801-811e-8c24402c90a7","Type":"ContainerStarted","Data":"98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f"}
Oct 30 22:51:05.991211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.991186    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" event={"ID":"5c07dbf8-219e-4107-b1c4-47a53bed6c7e","Type":"ContainerStarted","Data":"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"}
Oct 30 22:51:05.991333 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:05.991235    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" event={"ID":"5c07dbf8-219e-4107-b1c4-47a53bed6c7e","Type":"ContainerStarted","Data":"eeecc7d0839f246c0f4c3ca52992e7ebfc6181486db4b58fabe2137a97466cf2"}
Oct 30 22:51:06.111824 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.111764    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8" podStartSLOduration=2.111748043 podStartE2EDuration="2.111748043s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:06.110894916 +0000 UTC m=+4389.284219342" watchObservedRunningTime="2026-10-30 22:51:06.111748043 +0000 UTC m=+4389.285072472"
Oct 30 22:51:06.254543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.254471    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podStartSLOduration=2.2544513 podStartE2EDuration="2.2544513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:06.236117508 +0000 UTC m=+4389.409441935" watchObservedRunningTime="2026-10-30 22:51:06.2544513 +0000 UTC m=+4389.427775727"
Oct 30 22:51:06.600250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.600219    4882 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:06.602036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.602008    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 22:51:06.602165 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.602053    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 22:51:06.999636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.999534    4882 generic.go:356] "Generic (PLEG): container finished" podID="c1b6ceb6-68aa-4801-811e-8c24402c90a7" containerID="d9b62645d124d88d6c04331214e3cbb600b0eef003c9be15550661add739f832" exitCode=0
Oct 30 22:51:06.999636 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:06.999596    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8" event={"ID":"c1b6ceb6-68aa-4801-811e-8c24402c90a7","Type":"ContainerDied","Data":"d9b62645d124d88d6c04331214e3cbb600b0eef003c9be15550661add739f832"}
Oct 30 22:51:07.600387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:07.600348    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 22:51:07.600834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:07.600408    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 22:51:08.189000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.188974    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:08.277742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.277653    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\") pod \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") "
Oct 30 22:51:08.277742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.277693    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\") pod \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") "
Oct 30 22:51:08.277742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.277716    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\" (UniqueName: \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\") pod \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\" (UID: \"c1b6ceb6-68aa-4801-811e-8c24402c90a7\") "
Oct 30 22:51:08.278062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.278039    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca" pod "c1b6ceb6-68aa-4801-811e-8c24402c90a7" (UID: "c1b6ceb6-68aa-4801-811e-8c24402c90a7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:08.279993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.279964    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8" pod "c1b6ceb6-68aa-4801-811e-8c24402c90a7" (UID: "c1b6ceb6-68aa-4801-811e-8c24402c90a7"). InnerVolumeSpecName "kube-api-access-x5jk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:08.280137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.280045    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca" pod "c1b6ceb6-68aa-4801-811e-8c24402c90a7" (UID: "c1b6ceb6-68aa-4801-811e-8c24402c90a7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:08.379100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.379058    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:08.379100 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.379097    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c1b6ceb6-68aa-4801-811e-8c24402c90a7-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:08.379319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.379112    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5jk8\" (UniqueName: \"kubernetes.io/projected/c1b6ceb6-68aa-4801-811e-8c24402c90a7-kube-api-access-x5jk8\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:08.599815 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.599779    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 22:51:08.600025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.599851    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 22:51:08.962080 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.962001    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"]
Oct 30 22:51:08.967997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:08.967960    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-43d592-hgkhk"]
Oct 30 22:51:09.014011 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:09.013979    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"
Oct 30 22:51:09.014212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:09.014006    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8" event={"ID":"c1b6ceb6-68aa-4801-811e-8c24402c90a7","Type":"ContainerDied","Data":"98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f"}
Oct 30 22:51:09.014212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:09.014062    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98af7769002e606f81757e767bfedd57aec3a6f6797b145fe81d208b0513fc0f"
Oct 30 22:51:09.600661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:09.600624    4882 patch_prober.go:29] interesting pod/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused" start-of-body=
Oct 30 22:51:09.600834 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:09.600682    4882 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.1.98:15021/healthz/ready\": dial tcp 10.132.1.98:15021: connect: connection refused"
Oct 30 22:51:10.552235 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:10.552190    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:10.552811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:10.552420    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy" containerID="cri-o://7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0" gracePeriod=30
Oct 30 22:51:10.956959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:10.956926    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef68f459-d568-42fe-9f02-740b2f9a43eb" path="/var/lib/kubelet/pods/ef68f459-d568-42fe-9f02-740b2f9a43eb/volumes"
Oct 30 22:51:15.348786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:15.348749    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:51:15.554697 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:15.554657    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:51:16.771956 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.771918    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"]
Oct 30 22:51:16.772458 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.772444    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c1b6ceb6-68aa-4801-811e-8c24402c90a7" containerName="revoke-keys"
Oct 30 22:51:16.772510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.772459    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1b6ceb6-68aa-4801-811e-8c24402c90a7" containerName="revoke-keys"
Oct 30 22:51:16.772572 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.772563    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1b6ceb6-68aa-4801-811e-8c24402c90a7" containerName="revoke-keys"
Oct 30 22:51:16.776939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.776912    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.829194 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.829157    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"]
Oct 30 22:51:16.832962 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.832934    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:16.833202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.833178    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:51:16.874943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.874902    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v99f6\" (UniqueName: \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.875126 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.874956    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.875208 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.875181    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.976838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.976798    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.977025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.976902    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v99f6\" (UniqueName: \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.977025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.976937    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.977719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.977695    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:16.979379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:16.979359    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:17.039022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:17.038941    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v99f6\" (UniqueName: \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:17.091295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:17.091264    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:17.469129 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:17.469093    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98fcf6c9_f767_48b3_b861_bf4e5c56e43d.slice/crio-eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb WatchSource:0}: Error finding container eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb: Status 404 returned error can't find the container with id eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb
Oct 30 22:51:17.470721 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:17.470695    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"]
Oct 30 22:51:17.947754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:17.947725    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:18.090961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.090861    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.090961 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.090926    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.090974    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.090997    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091019    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091061    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091086    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091134    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091221 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091209    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:18.091664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091217    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\") pod \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\" (UID: \"5c07dbf8-219e-4107-b1c4-47a53bed6c7e\") "
Oct 30 22:51:18.091664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091490    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:18.091664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091569    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:18.091664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091572    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:18.091664 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091606    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:18.091939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091865    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.091939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091890    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.091939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091908    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.091939 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091923    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.092114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.091939    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.093525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.093486    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:51:18.093525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.093497    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:18.093928 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.093899    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "kube-api-access-lmd2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:18.094025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.093947    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy" pod "5c07dbf8-219e-4107-b1c4-47a53bed6c7e" (UID: "5c07dbf8-219e-4107-b1c4-47a53bed6c7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:51:18.101319 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.101288    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq" event={"ID":"98fcf6c9-f767-48b3-b861-bf4e5c56e43d","Type":"ContainerStarted","Data":"e5dd195ec7c7e8bba7ed376c351c969ac32ffac535b4f96b2af68d53785f2067"}
Oct 30 22:51:18.101467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.101335    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq" event={"ID":"98fcf6c9-f767-48b3-b861-bf4e5c56e43d","Type":"ContainerStarted","Data":"eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb"}
Oct 30 22:51:18.102960 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.102938    4882 generic.go:356] "Generic (PLEG): container finished" podID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerID="7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0" exitCode=0
Oct 30 22:51:18.103043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.102983    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"
Oct 30 22:51:18.103043 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.103018    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" event={"ID":"5c07dbf8-219e-4107-b1c4-47a53bed6c7e","Type":"ContainerDied","Data":"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"}
Oct 30 22:51:18.103150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.103081    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4" event={"ID":"5c07dbf8-219e-4107-b1c4-47a53bed6c7e","Type":"ContainerDied","Data":"eeecc7d0839f246c0f4c3ca52992e7ebfc6181486db4b58fabe2137a97466cf2"}
Oct 30 22:51:18.103150 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.103112    4882 scope.go:122] "RemoveContainer" containerID="7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"
Oct 30 22:51:18.121982 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.121953    4882 scope.go:122] "RemoveContainer" containerID="7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"
Oct 30 22:51:18.122352 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:18.122325    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0\": container with ID starting with 7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0 not found: ID does not exist" containerID="7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"
Oct 30 22:51:18.122450 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.122362    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0"} err="failed to get container status \"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0\": rpc error: code = NotFound desc = could not find container \"7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0\": container with ID starting with 7af781de784b602c0269bf97fc9aa7452ca58ef47f2745da082cf477934387f0 not found: ID does not exist"
Oct 30 22:51:18.193409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.193359    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmd2n\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-kube-api-access-lmd2n\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.193409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.193399    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.193409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.193413    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.193698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.193430    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5c07dbf8-219e-4107-b1c4-47a53bed6c7e-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:18.235258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.235218    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:18.239624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.239579    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-26743a-openshift-default-56b6f9f987-8t8b4"]
Oct 30 22:51:18.354662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.354529    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq" podStartSLOduration=2.354514769 podStartE2EDuration="2.354514769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:18.351897958 +0000 UTC m=+4401.525222384" watchObservedRunningTime="2026-10-30 22:51:18.354514769 +0000 UTC m=+4401.527839245"
Oct 30 22:51:18.942776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:18.942739    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" path="/var/lib/kubelet/pods/5c07dbf8-219e-4107-b1c4-47a53bed6c7e/volumes"
Oct 30 22:51:19.111785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:19.111750    4882 generic.go:356] "Generic (PLEG): container finished" podID="98fcf6c9-f767-48b3-b861-bf4e5c56e43d" containerID="e5dd195ec7c7e8bba7ed376c351c969ac32ffac535b4f96b2af68d53785f2067" exitCode=0
Oct 30 22:51:19.112240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:19.111830    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq" event={"ID":"98fcf6c9-f767-48b3-b861-bf4e5c56e43d","Type":"ContainerDied","Data":"e5dd195ec7c7e8bba7ed376c351c969ac32ffac535b4f96b2af68d53785f2067"}
Oct 30 22:51:20.266867 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.266836    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:20.349051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.349002    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing" containerID="cri-o://8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4" gracePeriod=25
Oct 30 22:51:20.416195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.416094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\" (UniqueName: \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\") pod \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") "
Oct 30 22:51:20.416195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.416187    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\") pod \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") "
Oct 30 22:51:20.416449 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.416240    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\") pod \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\" (UID: \"98fcf6c9-f767-48b3-b861-bf4e5c56e43d\") "
Oct 30 22:51:20.416582 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.416556    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca" pod "98fcf6c9-f767-48b3-b861-bf4e5c56e43d" (UID: "98fcf6c9-f767-48b3-b861-bf4e5c56e43d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:20.416861 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.416837    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:20.418372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.418339    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6" pod "98fcf6c9-f767-48b3-b861-bf4e5c56e43d" (UID: "98fcf6c9-f767-48b3-b861-bf4e5c56e43d"). InnerVolumeSpecName "kube-api-access-v99f6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:20.418494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.418416    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca" pod "98fcf6c9-f767-48b3-b861-bf4e5c56e43d" (UID: "98fcf6c9-f767-48b3-b861-bf4e5c56e43d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:20.517996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.517953    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:20.517996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.517993    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v99f6\" (UniqueName: \"kubernetes.io/projected/98fcf6c9-f767-48b3-b861-bf4e5c56e43d-kube-api-access-v99f6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:20.555331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:20.555255    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing" containerID="cri-o://ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb" gracePeriod=25
Oct 30 22:51:20.808430 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:20.808339    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:51:21.127732 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:21.127700    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"
Oct 30 22:51:21.127900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:21.127726    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq" event={"ID":"98fcf6c9-f767-48b3-b861-bf4e5c56e43d","Type":"ContainerDied","Data":"eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb"}
Oct 30 22:51:21.127900 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:21.127775    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eeaee55d0c7b722fc84772ed61c28ee9df92b5e421eee8d41f40638809bbf2cb"
Oct 30 22:51:23.935186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:23.935148    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:24.946444 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:24.946409    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:27.834560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:27.834519    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:51:27.885095 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:27.885060    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:51:29.818684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:29.818644    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:51:32.834890 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:32.834826    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing" containerID="cri-o://24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270" gracePeriod=25
Oct 30 22:51:32.885251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:32.885211    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"]
Oct 30 22:51:32.885538 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:32.885493    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing" containerID="cri-o://8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16" gracePeriod=25
Oct 30 22:51:32.934802 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:32.934769    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-ae7a1f-9nvdq"]
Oct 30 22:51:32.943249 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:32.943209    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98fcf6c9-f767-48b3-b861-bf4e5c56e43d" path="/var/lib/kubelet/pods/98fcf6c9-f767-48b3-b861-bf4e5c56e43d/volumes"
Oct 30 22:51:33.934998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:33.934958    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:34.946773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:34.946734    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:36.036365 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:36.036327    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:36.036814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:36.036326    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:40.686481 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.686448    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:51:40.739409 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.739375    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\") pod \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") "
Oct 30 22:51:40.739655 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.739474    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\" (UniqueName: \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\") pod \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\" (UID: \"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2\") "
Oct 30 22:51:40.739814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.739788    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume" pod "0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" (UID: "0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:40.739921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.739905    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:40.742628 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.742605    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt" pod "0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" (UID: "0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2"). InnerVolumeSpecName "kube-api-access-thfvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:40.793175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793144    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"]
Oct 30 22:51:40.793743 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793724    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy"
Oct 30 22:51:40.793862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793744    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy"
Oct 30 22:51:40.793862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793773    4882 cpu_manager.go:414] "containerMap: removing container" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing"
Oct 30 22:51:40.793862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793782    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing"
Oct 30 22:51:40.793862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793828    4882 cpu_manager.go:414] "containerMap: removing container" podUID="98fcf6c9-f767-48b3-b861-bf4e5c56e43d" containerName="revoke-keys"
Oct 30 22:51:40.793862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793846    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98fcf6c9-f767-48b3-b861-bf4e5c56e43d" containerName="revoke-keys"
Oct 30 22:51:40.794141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.793987    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c07dbf8-219e-4107-b1c4-47a53bed6c7e" containerName="istio-proxy"
Oct 30 22:51:40.794141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.794003    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerName="payload-processing"
Oct 30 22:51:40.794141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.794020    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="98fcf6c9-f767-48b3-b861-bf4e5c56e43d" containerName="revoke-keys"
Oct 30 22:51:40.797857 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.797827    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.799005 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.798975    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:51:40.820231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.820195    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"]
Oct 30 22:51:40.840518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.840480    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\" (UniqueName: \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\") pod \"b98d7f63-ed19-42f1-99fe-050591fe84eb\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") "
Oct 30 22:51:40.840748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.840726    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\") pod \"b98d7f63-ed19-42f1-99fe-050591fe84eb\" (UID: \"b98d7f63-ed19-42f1-99fe-050591fe84eb\") "
Oct 30 22:51:40.840950 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.840923    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.841152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.840972    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.841152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.841080    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume" pod "b98d7f63-ed19-42f1-99fe-050591fe84eb" (UID: "b98d7f63-ed19-42f1-99fe-050591fe84eb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:40.841251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.841206    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.841345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.841331    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-thfvt\" (UniqueName: \"kubernetes.io/projected/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2-kube-api-access-thfvt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:40.841393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.841347    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b98d7f63-ed19-42f1-99fe-050591fe84eb-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:40.842701 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.842677    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb" pod "b98d7f63-ed19-42f1-99fe-050591fe84eb" (UID: "b98d7f63-ed19-42f1-99fe-050591fe84eb"). InnerVolumeSpecName "kube-api-access-ghmnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:40.854392 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.854364    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:40.854558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.854408    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:51:40.942353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.942259    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.942353 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.942334    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.942630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.942371    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.942630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.942515    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghmnb\" (UniqueName: \"kubernetes.io/projected/b98d7f63-ed19-42f1-99fe-050591fe84eb-kube-api-access-ghmnb\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:40.942989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.942967    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:40.944791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:40.944764    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:41.004617 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.004573    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\") pod \"maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:41.121282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.121246    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:41.254880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.254788    4882 generic.go:356] "Generic (PLEG): container finished" podID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" containerID="8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4" exitCode=0
Oct 30 22:51:41.254880 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.254841    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"
Oct 30 22:51:41.255140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.254871    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" event={"ID":"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2","Type":"ContainerDied","Data":"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"}
Oct 30 22:51:41.255140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.254926    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k" event={"ID":"0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2","Type":"ContainerDied","Data":"a83940031090d8ac9c48e8a741056de2bf6555b08fa6840e2ff7c08f54cd46c4"}
Oct 30 22:51:41.255140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.254948    4882 scope.go:122] "RemoveContainer" containerID="8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"
Oct 30 22:51:41.256547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.256515    4882 generic.go:356] "Generic (PLEG): container finished" podID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerID="ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb" exitCode=0
Oct 30 22:51:41.256708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.256561    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"
Oct 30 22:51:41.256708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.256580    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" event={"ID":"b98d7f63-ed19-42f1-99fe-050591fe84eb","Type":"ContainerDied","Data":"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"}
Oct 30 22:51:41.256708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.256656    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5" event={"ID":"b98d7f63-ed19-42f1-99fe-050591fe84eb","Type":"ContainerDied","Data":"bc0ab01363552bc9bdfc23e3afdba079d6f2cac5958013b7582a14341ed14e3e"}
Oct 30 22:51:41.265153 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.265131    4882 scope.go:122] "RemoveContainer" containerID="8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"
Oct 30 22:51:41.265480 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:41.265460    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4\": container with ID starting with 8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4 not found: ID does not exist" containerID="8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"
Oct 30 22:51:41.265568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.265491    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4"} err="failed to get container status \"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4\": rpc error: code = NotFound desc = could not find container \"8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4\": container with ID starting with 8a3c6fd1215b231edcf70653af17044c683e5cda58de4edd9a99e32490e802a4 not found: ID does not exist"
Oct 30 22:51:41.265568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.265515    4882 scope.go:122] "RemoveContainer" containerID="ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"
Oct 30 22:51:41.274828 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.274809    4882 scope.go:122] "RemoveContainer" containerID="ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"
Oct 30 22:51:41.275098 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:41.275076    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb\": container with ID starting with ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb not found: ID does not exist" containerID="ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"
Oct 30 22:51:41.275181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.275111    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb"} err="failed to get container status \"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb\": rpc error: code = NotFound desc = could not find container \"ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb\": container with ID starting with ae25fd2c21337787f2bf544a780e7807d4df72a330e9851cfdfa10e1209162bb not found: ID does not exist"
Oct 30 22:51:41.422988 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.422946    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:51:41.428087 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.428056    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-5bb0b1-6cb99c7fd7-vl47k"]
Oct 30 22:51:41.526468 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:41.526433    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8170c4a_e7eb_4e0f_bda0_a2f367024172.slice/crio-9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46 WatchSource:0}: Error finding container 9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46: Status 404 returned error can't find the container with id 9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46
Oct 30 22:51:41.529505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.529484    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"]
Oct 30 22:51:41.539337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.539294    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:51:41.543301 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:41.543272    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-5bb0b1-84fcf8f4c9-nrgq5"]
Oct 30 22:51:42.274204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:42.274094    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv" event={"ID":"d8170c4a-e7eb-4e0f-bda0-a2f367024172","Type":"ContainerStarted","Data":"d0efa4ad33b6796d1efd8a240fbbe87033041c041c64a9db58122d49cc8bd69b"}
Oct 30 22:51:42.274204 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:42.274190    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv" event={"ID":"d8170c4a-e7eb-4e0f-bda0-a2f367024172","Type":"ContainerStarted","Data":"9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46"}
Oct 30 22:51:42.391978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:42.391922    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv" podStartSLOduration=2.391906874 podStartE2EDuration="2.391906874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:42.39055029 +0000 UTC m=+4425.563874714" watchObservedRunningTime="2026-10-30 22:51:42.391906874 +0000 UTC m=+4425.565231302"
Oct 30 22:51:42.943062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:42.943027    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2" path="/var/lib/kubelet/pods/0ae872ca-1d79-4b3e-95e0-8c38f0fe0db2/volumes"
Oct 30 22:51:42.943446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:42.943432    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" path="/var/lib/kubelet/pods/b98d7f63-ed19-42f1-99fe-050591fe84eb/volumes"
Oct 30 22:51:43.284563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:43.284466    4882 generic.go:356] "Generic (PLEG): container finished" podID="d8170c4a-e7eb-4e0f-bda0-a2f367024172" containerID="d0efa4ad33b6796d1efd8a240fbbe87033041c041c64a9db58122d49cc8bd69b" exitCode=0
Oct 30 22:51:43.285038 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:43.284554    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv" event={"ID":"d8170c4a-e7eb-4e0f-bda0-a2f367024172","Type":"ContainerDied","Data":"d0efa4ad33b6796d1efd8a240fbbe87033041c041c64a9db58122d49cc8bd69b"}
Oct 30 22:51:44.427649 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.427617    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:44.481289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.481257    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\") pod \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") "
Oct 30 22:51:44.481289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.481294    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\") pod \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") "
Oct 30 22:51:44.481519 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.481350    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\") pod \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\" (UID: \"d8170c4a-e7eb-4e0f-bda0-a2f367024172\") "
Oct 30 22:51:44.481839 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.481806    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca" pod "d8170c4a-e7eb-4e0f-bda0-a2f367024172" (UID: "d8170c4a-e7eb-4e0f-bda0-a2f367024172"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:44.483522 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.483499    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca" pod "d8170c4a-e7eb-4e0f-bda0-a2f367024172" (UID: "d8170c4a-e7eb-4e0f-bda0-a2f367024172"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:51:44.483624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.483518    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r" pod "d8170c4a-e7eb-4e0f-bda0-a2f367024172" (UID: "d8170c4a-e7eb-4e0f-bda0-a2f367024172"). InnerVolumeSpecName "kube-api-access-2p82r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:44.582937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.582902    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p82r\" (UniqueName: \"kubernetes.io/projected/d8170c4a-e7eb-4e0f-bda0-a2f367024172-kube-api-access-2p82r\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:44.582937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.582937    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:44.583142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:44.582950    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8170c4a-e7eb-4e0f-bda0-a2f367024172-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:45.248932 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:45.248832    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 22:51:45.290682 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:45.290642    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 22:51:45.303467 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:45.303432    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv" event={"ID":"d8170c4a-e7eb-4e0f-bda0-a2f367024172","Type":"ContainerDied","Data":"9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46"}
Oct 30 22:51:45.303661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:45.303494    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9c1b52948d037809f9b20cbc909b6be02cea99b0d1b3c6039ba16937ab942a46"
Oct 30 22:51:45.303661 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:45.303450    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"
Oct 30 22:51:46.036264 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:46.036228    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:46.036716 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:46.036289    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:51:50.809140 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:50.809103    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:51:53.153916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.153887    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:51:53.181934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.181910    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:51:53.300934 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.300902    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\") pod \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") "
Oct 30 22:51:53.301127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.300949    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\") pod \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") "
Oct 30 22:51:53.301127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301052    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\" (UniqueName: \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\") pod \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\" (UID: \"aa6c992a-572e-4fb4-a5d0-3bba651d05f7\") "
Oct 30 22:51:53.301127 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301082    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\" (UniqueName: \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\") pod \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\" (UID: \"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96\") "
Oct 30 22:51:53.301374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301294    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume" pod "89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" (UID: "89517b8e-a1d9-47c3-a2c3-a2d3f445ed96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:53.301443 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301386    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume" pod "aa6c992a-572e-4fb4-a5d0-3bba651d05f7" (UID: "aa6c992a-572e-4fb4-a5d0-3bba651d05f7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:51:53.301526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301510    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:53.301626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.301527    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:53.303404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.303379    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt" pod "aa6c992a-572e-4fb4-a5d0-3bba651d05f7" (UID: "aa6c992a-572e-4fb4-a5d0-3bba651d05f7"). InnerVolumeSpecName "kube-api-access-8wrbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:53.303521 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.303420    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t" pod "89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" (UID: "89517b8e-a1d9-47c3-a2c3-a2d3f445ed96"). InnerVolumeSpecName "kube-api-access-25p6t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:51:53.355940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.355853    4882 generic.go:356] "Generic (PLEG): container finished" podID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerID="24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270" exitCode=0
Oct 30 22:51:53.355940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.355908    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"
Oct 30 22:51:53.356140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.355936    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" event={"ID":"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96","Type":"ContainerDied","Data":"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"}
Oct 30 22:51:53.356140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.355996    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl" event={"ID":"89517b8e-a1d9-47c3-a2c3-a2d3f445ed96","Type":"ContainerDied","Data":"86ff5b29e0e9805c7425f64667d656c4bc856f9535704f4bfbaaa1b526be9cf7"}
Oct 30 22:51:53.356140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.356028    4882 scope.go:122] "RemoveContainer" containerID="24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"
Oct 30 22:51:53.357482 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.357458    4882 generic.go:356] "Generic (PLEG): container finished" podID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerID="8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16" exitCode=0
Oct 30 22:51:53.357705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.357497    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"
Oct 30 22:51:53.357705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.357568    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" event={"ID":"aa6c992a-572e-4fb4-a5d0-3bba651d05f7","Type":"ContainerDied","Data":"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"}
Oct 30 22:51:53.357705 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.357627    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk" event={"ID":"aa6c992a-572e-4fb4-a5d0-3bba651d05f7","Type":"ContainerDied","Data":"1740e97538a903511f3d1a200dacee3282cd42fce2ecb0f255b30b9d2b98c704"}
Oct 30 22:51:53.367951 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.367929    4882 scope.go:122] "RemoveContainer" containerID="24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"
Oct 30 22:51:53.368259 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:53.368235    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270\": container with ID starting with 24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270 not found: ID does not exist" containerID="24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"
Oct 30 22:51:53.368355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.368264    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270"} err="failed to get container status \"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270\": rpc error: code = NotFound desc = could not find container \"24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270\": container with ID starting with 24a35a9c3ea3489af5b7c288b3f6cd9f1b63f01ac08ea2ca3751155452e84270 not found: ID does not exist"
Oct 30 22:51:53.368355 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.368283    4882 scope.go:122] "RemoveContainer" containerID="8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"
Oct 30 22:51:53.378030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.378013    4882 scope.go:122] "RemoveContainer" containerID="8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"
Oct 30 22:51:53.378326 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:51:53.378305    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16\": container with ID starting with 8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16 not found: ID does not exist" containerID="8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"
Oct 30 22:51:53.378379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.378335    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16"} err="failed to get container status \"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16\": rpc error: code = NotFound desc = could not find container \"8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16\": container with ID starting with 8fd16cb77b60beaeea04ade96e5d4089ef3ce7490f5ea44a8f51145b89cc2e16 not found: ID does not exist"
Oct 30 22:51:53.403437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.403400    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wrbt\" (UniqueName: \"kubernetes.io/projected/aa6c992a-572e-4fb4-a5d0-3bba651d05f7-kube-api-access-8wrbt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:53.403611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.403439    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-25p6t\" (UniqueName: \"kubernetes.io/projected/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96-kube-api-access-25p6t\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:51:53.508711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.508674    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:51:53.513496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.513467    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-ae7a1f-fb544d65f-xm9pl"]
Oct 30 22:51:53.562022 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.561991    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:51:53.639248 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.639164    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:51:53.657155 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:53.657117    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-ae7a1f-76f66786c9-mg2pk"]
Oct 30 22:51:54.257234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:54.257201    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"]
Oct 30 22:51:54.259833 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:54.259811    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-5bb0b1-pczw8"]
Oct 30 22:51:54.942864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:54.942823    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" path="/var/lib/kubelet/pods/89517b8e-a1d9-47c3-a2c3-a2d3f445ed96/volumes"
Oct 30 22:51:54.943332 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:54.943310    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" path="/var/lib/kubelet/pods/aa6c992a-572e-4fb4-a5d0-3bba651d05f7/volumes"
Oct 30 22:51:54.943819 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:54.943798    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1b6ceb6-68aa-4801-811e-8c24402c90a7" path="/var/lib/kubelet/pods/c1b6ceb6-68aa-4801-811e-8c24402c90a7/volumes"
Oct 30 22:51:58.207324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207285    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"]
Oct 30 22:51:58.207838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207824    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d8170c4a-e7eb-4e0f-bda0-a2f367024172" containerName="revoke-keys"
Oct 30 22:51:58.207883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207839    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8170c4a-e7eb-4e0f-bda0-a2f367024172" containerName="revoke-keys"
Oct 30 22:51:58.207883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207860    4882 cpu_manager.go:414] "containerMap: removing container" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing"
Oct 30 22:51:58.207883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207866    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing"
Oct 30 22:51:58.207883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207878    4882 cpu_manager.go:414] "containerMap: removing container" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207883    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207896    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207901    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207967    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa6c992a-572e-4fb4-a5d0-3bba651d05f7" containerName="payload-pre-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207977    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="89517b8e-a1d9-47c3-a2c3-a2d3f445ed96" containerName="payload-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207983    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b98d7f63-ed19-42f1-99fe-050591fe84eb" containerName="payload-pre-processing"
Oct 30 22:51:58.208019 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.207995    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8170c4a-e7eb-4e0f-bda0-a2f367024172" containerName="revoke-keys"
Oct 30 22:51:58.212812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.212789    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.248190 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.248153    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.248372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.248286    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lc58t\" (UniqueName: \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.248372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.248353    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.265186 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.265155    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"]
Oct 30 22:51:58.269903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.269871    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"]
Oct 30 22:51:58.270089 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.270007    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.312506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.312471    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:51:58.312506 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.312480    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:51:58.323812 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.323774    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"]
Oct 30 22:51:58.349775 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.349736    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.349981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.349797    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.349981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.349844    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.349981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.349867    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lc58t\" (UniqueName: \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.349981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.349974    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvdh\" (UniqueName: \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.350156 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.350007    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.350659 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.350634    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.352330 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.352309    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.412829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.412786    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lc58t\" (UniqueName: \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\") pod \"maas-api-revoke-sub-a1a785c4-x66bt\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.450598 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.450560    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvdh\" (UniqueName: \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.450755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.450711    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.450755 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.450744    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.451406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.451383    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.453071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.453050    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.512789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.512709    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvdh\" (UniqueName: \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\") pod \"maas-api-revoke-sub-82285251-k586m\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.526525 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.526497    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:51:58.584133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.584094    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:51:58.905625 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:58.905577    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod450d4462_660e_4ff3_849f_817528a4ace5.slice/crio-7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f WatchSource:0}: Error finding container 7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f: Status 404 returned error can't find the container with id 7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f
Oct 30 22:51:58.907696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:58.907671    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"]
Oct 30 22:51:59.001022 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:51:59.000977    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3da63bcb_26b5_4a4f_820d_b85a44f2f3d0.slice/crio-02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9 WatchSource:0}: Error finding container 02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9: Status 404 returned error can't find the container with id 02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9
Oct 30 22:51:59.003648 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.003620    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"]
Oct 30 22:51:59.399683 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.399638    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m" event={"ID":"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0","Type":"ContainerStarted","Data":"5d919b0deb83285070d6e5633878f5e1fde0896bac72ff3e4ee41c10998b72b1"}
Oct 30 22:51:59.400175 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.399694    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m" event={"ID":"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0","Type":"ContainerStarted","Data":"02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9"}
Oct 30 22:51:59.401422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.401395    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt" event={"ID":"450d4462-660e-4ff3-849f-817528a4ace5","Type":"ContainerStarted","Data":"9a6c12b5f72ad55a8a08d7281f9fb5a4434c35694226ef0a8872aceeae0479dc"}
Oct 30 22:51:59.401577 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.401437    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt" event={"ID":"450d4462-660e-4ff3-849f-817528a4ace5","Type":"ContainerStarted","Data":"7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f"}
Oct 30 22:51:59.519952 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.519850    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m" podStartSLOduration=1.519833481 podStartE2EDuration="1.519833481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:59.517345021 +0000 UTC m=+4442.690669447" watchObservedRunningTime="2026-10-30 22:51:59.519833481 +0000 UTC m=+4442.693157908"
Oct 30 22:51:59.636518 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:51:59.636459    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt" podStartSLOduration=1.636442496 podStartE2EDuration="1.636442496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:51:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:51:59.633370552 +0000 UTC m=+4442.806694979" watchObservedRunningTime="2026-10-30 22:51:59.636442496 +0000 UTC m=+4442.809766923"
Oct 30 22:52:00.269914 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.269785    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main" containerID="cri-o://8e1e41e2e64ddfa0a43b18d38aee9bde41f4e9b7861df910190c41378528c2f8" gracePeriod=45
Oct 30 22:52:00.314885 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.314842    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main" containerID="cri-o://56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276" gracePeriod=45
Oct 30 22:52:00.410765 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.410727    4882 generic.go:356] "Generic (PLEG): container finished" podID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerID="8e1e41e2e64ddfa0a43b18d38aee9bde41f4e9b7861df910190c41378528c2f8" exitCode=0
Oct 30 22:52:00.411185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.410794    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" event={"ID":"b8c052ee-942c-4e66-a385-12e5f90f16e3","Type":"ContainerDied","Data":"8e1e41e2e64ddfa0a43b18d38aee9bde41f4e9b7861df910190c41378528c2f8"}
Oct 30 22:52:00.415970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.415938    4882 generic.go:356] "Generic (PLEG): container finished" podID="3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" containerID="5d919b0deb83285070d6e5633878f5e1fde0896bac72ff3e4ee41c10998b72b1" exitCode=0
Oct 30 22:52:00.416152 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.416020    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m" event={"ID":"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0","Type":"ContainerDied","Data":"5d919b0deb83285070d6e5633878f5e1fde0896bac72ff3e4ee41c10998b72b1"}
Oct 30 22:52:00.418065 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.418037    4882 generic.go:356] "Generic (PLEG): container finished" podID="450d4462-660e-4ff3-849f-817528a4ace5" containerID="9a6c12b5f72ad55a8a08d7281f9fb5a4434c35694226ef0a8872aceeae0479dc" exitCode=0
Oct 30 22:52:00.418215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.418074    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt" event={"ID":"450d4462-660e-4ff3-849f-817528a4ace5","Type":"ContainerDied","Data":"9a6c12b5f72ad55a8a08d7281f9fb5a4434c35694226ef0a8872aceeae0479dc"}
Oct 30 22:52:00.547106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.547028    4882 scope.go:122] "RemoveContainer" containerID="c428ce29530cda277451ef560741fb49be1a66d2a6400ffed35579c1f5fddfe8"
Oct 30 22:52:00.596870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.596842    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 22:52:00.629871 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.629845    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 22:52:00.678852 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.678818    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\" (UniqueName: \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.678870    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.678900    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.678926    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\" (UniqueName: \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.678964    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679078 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679028    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679124    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679190    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679202    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679227    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679274    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679306    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679362    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\" (UniqueName: \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\") pod \"b8c052ee-942c-4e66-a385-12e5f90f16e3\" (UID: \"b8c052ee-942c-4e66-a385-12e5f90f16e3\") "
Oct 30 22:52:00.679786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679410    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.679786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679422    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.679786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679430    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\") pod \"c83340a0-e2a0-4abc-87fd-58889e90a911\" (UID: \"c83340a0-e2a0-4abc-87fd-58889e90a911\") "
Oct 30 22:52:00.679954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679838    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.679954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679847    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.679954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679886    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.679954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679903    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.679954 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.679919    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.680213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.680056    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.681931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.681898    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:00.682074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.681989    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "kube-api-access-kbpgg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:00.682142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.682105    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:00.682142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.682123    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "kube-api-access-vthgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:00.682351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.682333    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm" pod "c83340a0-e2a0-4abc-87fd-58889e90a911" (UID: "c83340a0-e2a0-4abc-87fd-58889e90a911"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.683759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.683738    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm" pod "b8c052ee-942c-4e66-a385-12e5f90f16e3" (UID: "b8c052ee-942c-4e66-a385-12e5f90f16e3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:00.781800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781759    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b8c052ee-942c-4e66-a385-12e5f90f16e3-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.781800 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781798    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b8c052ee-942c-4e66-a385-12e5f90f16e3-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781811    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781827    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vthgr\" (UniqueName: \"kubernetes.io/projected/b8c052ee-942c-4e66-a385-12e5f90f16e3-kube-api-access-vthgr\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781851    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781865    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c83340a0-e2a0-4abc-87fd-58889e90a911-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781877    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbpgg\" (UniqueName: \"kubernetes.io/projected/c83340a0-e2a0-4abc-87fd-58889e90a911-kube-api-access-kbpgg\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:00.782075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:00.781893    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c83340a0-e2a0-4abc-87fd-58889e90a911-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.425761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.425734    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"
Oct 30 22:52:01.426283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.425773    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r" event={"ID":"b8c052ee-942c-4e66-a385-12e5f90f16e3","Type":"ContainerDied","Data":"115b8646179bb5d76464af99b0f0f2040fb3fabe087e0500b9129c2c801e9d26"}
Oct 30 22:52:01.426283 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.425826    4882 scope.go:122] "RemoveContainer" containerID="8e1e41e2e64ddfa0a43b18d38aee9bde41f4e9b7861df910190c41378528c2f8"
Oct 30 22:52:01.427520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.427492    4882 generic.go:356] "Generic (PLEG): container finished" podID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerID="56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276" exitCode=0
Oct 30 22:52:01.427662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.427562    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"
Oct 30 22:52:01.427662 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.427631    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" event={"ID":"c83340a0-e2a0-4abc-87fd-58889e90a911","Type":"ContainerDied","Data":"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"}
Oct 30 22:52:01.427798 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.427666    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff" event={"ID":"c83340a0-e2a0-4abc-87fd-58889e90a911","Type":"ContainerDied","Data":"f168316b2e8e71900e277759373f5809af4db4434ffaf7a6443f675a66fc6717"}
Oct 30 22:52:01.436847 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.436817    4882 scope.go:122] "RemoveContainer" containerID="56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"
Oct 30 22:52:01.449404 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.449376    4882 scope.go:122] "RemoveContainer" containerID="56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"
Oct 30 22:52:01.449872 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:52:01.449829    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276\": container with ID starting with 56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276 not found: ID does not exist" containerID="56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"
Oct 30 22:52:01.449957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.449886    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276"} err="failed to get container status \"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276\": rpc error: code = NotFound desc = could not find container \"56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276\": container with ID starting with 56ce5ade8b7f5bf968f18f82bc7fa3fee1eace40b6a048b847178a10832bd276 not found: ID does not exist"
Oct 30 22:52:01.560730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.560702    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:52:01.590838 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.590807    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\") pod \"450d4462-660e-4ff3-849f-817528a4ace5\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") "
Oct 30 22:52:01.590979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.590916    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\" (UniqueName: \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\") pod \"450d4462-660e-4ff3-849f-817528a4ace5\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") "
Oct 30 22:52:01.591062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.591048    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\") pod \"450d4462-660e-4ff3-849f-817528a4ace5\" (UID: \"450d4462-660e-4ff3-849f-817528a4ace5\") "
Oct 30 22:52:01.591161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.591138    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca" pod "450d4462-660e-4ff3-849f-817528a4ace5" (UID: "450d4462-660e-4ff3-849f-817528a4ace5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:01.591372 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.591354    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.593113 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.593088    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t" pod "450d4462-660e-4ff3-849f-817528a4ace5" (UID: "450d4462-660e-4ff3-849f-817528a4ace5"). InnerVolumeSpecName "kube-api-access-lc58t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:01.593218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.593130    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca" pod "450d4462-660e-4ff3-849f-817528a4ace5" (UID: "450d4462-660e-4ff3-849f-817528a4ace5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:01.596740 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.596716    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 22:52:01.611718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.611686    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-2-w3-4d14e0-kserve-5ff56b94d5-9f28r"]
Oct 30 22:52:01.618151 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.618128    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:52:01.692626 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.692515    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\") pod \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") "
Oct 30 22:52:01.692864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.692753    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\") pod \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") "
Oct 30 22:52:01.692864 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.692806    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\" (UniqueName: \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\") pod \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\" (UID: \"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0\") "
Oct 30 22:52:01.693162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.693138    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lc58t\" (UniqueName: \"kubernetes.io/projected/450d4462-660e-4ff3-849f-817528a4ace5-kube-api-access-lc58t\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.693162 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.693163    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/450d4462-660e-4ff3-849f-817528a4ace5-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.693324 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.693140    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca" pod "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" (UID: "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:01.694957 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.694932    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca" pod "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" (UID: "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:01.695161 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.695140    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh" pod "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" (UID: "3da63bcb-26b5-4a4f-820d-b85a44f2f3d0"). InnerVolumeSpecName "kube-api-access-9nvdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:01.741955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.741916    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 22:52:01.744728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.744696    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-4d14e0/distinct-w3-4d14e0-kserve-768fcfdc4b-zczff"]
Oct 30 22:52:01.794720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.794669    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.794720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.794713    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nvdh\" (UniqueName: \"kubernetes.io/projected/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-kube-api-access-9nvdh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:01.794720 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:01.794726    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:02.436813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.436780    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"
Oct 30 22:52:02.437297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.436815    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt" event={"ID":"450d4462-660e-4ff3-849f-817528a4ace5","Type":"ContainerDied","Data":"7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f"}
Oct 30 22:52:02.437297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.436873    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7001c1f3c2dc5da642888b5155537c7ff18856e8dc68030f04f2d29a4ededf4f"
Oct 30 22:52:02.440854 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.440822    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m" event={"ID":"3da63bcb-26b5-4a4f-820d-b85a44f2f3d0","Type":"ContainerDied","Data":"02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9"}
Oct 30 22:52:02.441045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.440873    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02998c52513dca9632422a31e1abf30542753d0e788d5c1565467e6e7ae0dca9"
Oct 30 22:52:02.441045 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.440832    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"
Oct 30 22:52:02.942981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.942942    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" path="/var/lib/kubelet/pods/b8c052ee-942c-4e66-a385-12e5f90f16e3/volumes"
Oct 30 22:52:02.943486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:02.943465    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" path="/var/lib/kubelet/pods/c83340a0-e2a0-4abc-87fd-58889e90a911/volumes"
Oct 30 22:52:04.215955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.215915    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"]
Oct 30 22:52:04.216730 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216713    4882 cpu_manager.go:414] "containerMap: removing container" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main"
Oct 30 22:52:04.216790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216733    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main"
Oct 30 22:52:04.216790 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216778    4882 cpu_manager.go:414] "containerMap: removing container" podUID="3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" containerName="revoke-keys"
Oct 30 22:52:04.216859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216787    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" containerName="revoke-keys"
Oct 30 22:52:04.216859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216820    4882 cpu_manager.go:414] "containerMap: removing container" podUID="450d4462-660e-4ff3-849f-817528a4ace5" containerName="revoke-keys"
Oct 30 22:52:04.216859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216829    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="450d4462-660e-4ff3-849f-817528a4ace5" containerName="revoke-keys"
Oct 30 22:52:04.216859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216846    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main"
Oct 30 22:52:04.216859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216854    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main"
Oct 30 22:52:04.217028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216956    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="450d4462-660e-4ff3-849f-817528a4ace5" containerName="revoke-keys"
Oct 30 22:52:04.217028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216970    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c83340a0-e2a0-4abc-87fd-58889e90a911" containerName="main"
Oct 30 22:52:04.217028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216983    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8c052ee-942c-4e66-a385-12e5f90f16e3" containerName="main"
Oct 30 22:52:04.217028 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.216998    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" containerName="revoke-keys"
Oct 30 22:52:04.221559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.221536    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.275915 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.275881    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:04.276214 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.276189    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:52:04.322423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.322383    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.322423 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.322426    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnp8f\" (UniqueName: \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.322692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.322544    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.329500 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.329470    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"]
Oct 30 22:52:04.423959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.423916    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.423959 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.423963    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnp8f\" (UniqueName: \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.424260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.424008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.424692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.424671    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.426393 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.426368    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.486399 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.486330    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnp8f\" (UniqueName: \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\") pod \"maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.540323 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.540282    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:04.937413 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:52:04.937380    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod561e0967_db90_4d98_b691_86621581dbed.slice/crio-7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da WatchSource:0}: Error finding container 7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da: Status 404 returned error can't find the container with id 7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da
Oct 30 22:52:04.943937 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:04.943908    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"]
Oct 30 22:52:05.462438 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:05.462400    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc" event={"ID":"561e0967-db90-4d98-b691-86621581dbed","Type":"ContainerStarted","Data":"672c212c428661ea8d62fb8c9efdb6a11af73763cb4d588c480f7901eb40e7d6"}
Oct 30 22:52:05.462895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:05.462460    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc" event={"ID":"561e0967-db90-4d98-b691-86621581dbed","Type":"ContainerStarted","Data":"7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da"}
Oct 30 22:52:05.582474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:05.582425    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc" podStartSLOduration=1.582409524 podStartE2EDuration="1.582409524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:52:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:52:05.579167748 +0000 UTC m=+4448.752492175" watchObservedRunningTime="2026-10-30 22:52:05.582409524 +0000 UTC m=+4448.755733950"
Oct 30 22:52:06.470949 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:06.470907    4882 generic.go:356] "Generic (PLEG): container finished" podID="561e0967-db90-4d98-b691-86621581dbed" containerID="672c212c428661ea8d62fb8c9efdb6a11af73763cb4d588c480f7901eb40e7d6" exitCode=0
Oct 30 22:52:06.471442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:06.470989    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc" event={"ID":"561e0967-db90-4d98-b691-86621581dbed","Type":"ContainerDied","Data":"672c212c428661ea8d62fb8c9efdb6a11af73763cb4d588c480f7901eb40e7d6"}
Oct 30 22:52:07.624646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.624613    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:07.760297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.760208    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\" (UniqueName: \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\") pod \"561e0967-db90-4d98-b691-86621581dbed\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") "
Oct 30 22:52:07.760297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.760268    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\") pod \"561e0967-db90-4d98-b691-86621581dbed\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") "
Oct 30 22:52:07.760297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.760295    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\") pod \"561e0967-db90-4d98-b691-86621581dbed\" (UID: \"561e0967-db90-4d98-b691-86621581dbed\") "
Oct 30 22:52:07.760781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.760748    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca" pod "561e0967-db90-4d98-b691-86621581dbed" (UID: "561e0967-db90-4d98-b691-86621581dbed"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:07.762543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.762511    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca" pod "561e0967-db90-4d98-b691-86621581dbed" (UID: "561e0967-db90-4d98-b691-86621581dbed"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:07.762684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.762603    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f" pod "561e0967-db90-4d98-b691-86621581dbed" (UID: "561e0967-db90-4d98-b691-86621581dbed"). InnerVolumeSpecName "kube-api-access-fnp8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:07.861771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.861725    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnp8f\" (UniqueName: \"kubernetes.io/projected/561e0967-db90-4d98-b691-86621581dbed-kube-api-access-fnp8f\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:07.861771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.861761    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/561e0967-db90-4d98-b691-86621581dbed-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:07.861771 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:07.861776    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/561e0967-db90-4d98-b691-86621581dbed-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:08.485859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:08.485824    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"
Oct 30 22:52:08.486050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:08.485823    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc" event={"ID":"561e0967-db90-4d98-b691-86621581dbed","Type":"ContainerDied","Data":"7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da"}
Oct 30 22:52:08.486050 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:08.485952    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7e30ede114ae95399b4022a57adba1f5d3439b1142e477cf3eb26244969381da"
Oct 30 22:52:08.658490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:08.658445    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 22:52:08.679901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:08.679862    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 22:52:20.809292 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:52:20.809254    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:52:23.679813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.679763    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" containerID="cri-o://05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd" gracePeriod=45
Oct 30 22:52:23.704156 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.704111    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" containerID="cri-o://a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6" gracePeriod=45
Oct 30 22:52:23.798371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.798328    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main" probeResult="failure" output="Get \"https://10.132.0.65:8000/ready\": dial tcp 10.132.0.65:8000: connect: connection refused"
Oct 30 22:52:23.815609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.815550    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main" probeResult="failure" output="Get \"https://10.132.0.50:8000/ready\": dial tcp 10.132.0.50:8000: connect: connection refused"
Oct 30 22:52:23.975233 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.975205    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 22:52:23.993807 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:23.993781    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 22:52:24.132619 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132565    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.132831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132639    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\" (UniqueName: \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.132831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132675    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\" (UniqueName: \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.132831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132740    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.132831 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132811    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132861    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132862    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132904    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\" (UniqueName: \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132932    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\" (UniqueName: \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.132967    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\") pod \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\" (UID: \"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133009    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.133086 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133040    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.133502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133099    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\") pod \"28c383d2-e556-40d6-a361-2291d3e2613d\" (UID: \"28c383d2-e556-40d6-a361-2291d3e2613d\") "
Oct 30 22:52:24.133502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133122    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.133502 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133391    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.133712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133621    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.133712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133637    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.133712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133642    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.133712 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133650    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.134007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133860    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.134007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.133989    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.135789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.135765    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.135921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.135827    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:24.136082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.136055    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:52:24.136302 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.136268    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "kube-api-access-h8wtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:24.136832 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.136798    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs" pod "28c383d2-e556-40d6-a361-2291d3e2613d" (UID: "28c383d2-e556-40d6-a361-2291d3e2613d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:24.137273 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.137249    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79" pod "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" (UID: "186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16"). InnerVolumeSpecName "kube-api-access-qfd79". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234721    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234751    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8wtt\" (UniqueName: \"kubernetes.io/projected/28c383d2-e556-40d6-a361-2291d3e2613d-kube-api-access-h8wtt\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234762    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/28c383d2-e556-40d6-a361-2291d3e2613d-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234772    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234781    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234791    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.234799 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234799    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfd79\" (UniqueName: \"kubernetes.io/projected/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-kube-api-access-qfd79\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.235142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234808    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.235142 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.234817    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/28c383d2-e556-40d6-a361-2291d3e2613d-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:24.587916 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.587878    4882 generic.go:356] "Generic (PLEG): container finished" podID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerID="05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd" exitCode=0
Oct 30 22:52:24.588133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.587923    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"
Oct 30 22:52:24.588133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.587960    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" event={"ID":"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16","Type":"ContainerDied","Data":"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"}
Oct 30 22:52:24.588133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.588006    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr" event={"ID":"186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16","Type":"ContainerDied","Data":"9ee2158a9e09213d51217bd12207818a2208b62247a8bfefbb63b78d7d170878"}
Oct 30 22:52:24.588133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.588024    4882 scope.go:122] "RemoveContainer" containerID="05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"
Oct 30 22:52:24.589892 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.589866    4882 generic.go:356] "Generic (PLEG): container finished" podID="28c383d2-e556-40d6-a361-2291d3e2613d" containerID="a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6" exitCode=0
Oct 30 22:52:24.590007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.589926    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" event={"ID":"28c383d2-e556-40d6-a361-2291d3e2613d","Type":"ContainerDied","Data":"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"}
Oct 30 22:52:24.590007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.589933    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"
Oct 30 22:52:24.590007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.589957    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt" event={"ID":"28c383d2-e556-40d6-a361-2291d3e2613d","Type":"ContainerDied","Data":"b7af354acabe4719b10ea59ea01158393f2fa9fc7ab501a242d9c883940d61da"}
Oct 30 22:52:24.598772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.598750    4882 scope.go:122] "RemoveContainer" containerID="05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"
Oct 30 22:52:24.599069 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:52:24.599051    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd\": container with ID starting with 05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd not found: ID does not exist" containerID="05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"
Oct 30 22:52:24.599140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.599078    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd"} err="failed to get container status \"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd\": rpc error: code = NotFound desc = could not find container \"05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd\": container with ID starting with 05cf26a5e7479b1bddad2fd214020c3d11c92c9da59002fe34cf09ea0f491acd not found: ID does not exist"
Oct 30 22:52:24.599140 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.599096    4882 scope.go:122] "RemoveContainer" containerID="a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"
Oct 30 22:52:24.608202 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.608175    4882 scope.go:122] "RemoveContainer" containerID="a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"
Oct 30 22:52:24.608502 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:52:24.608483    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6\": container with ID starting with a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6 not found: ID does not exist" containerID="a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"
Oct 30 22:52:24.608550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.608512    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6"} err="failed to get container status \"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6\": rpc error: code = NotFound desc = could not find container \"a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6\": container with ID starting with a09ed56dba75edffa375c2c6531f0ead567b17108e5b532f692e83368879e8b6 not found: ID does not exist"
Oct 30 22:52:24.754448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.754412    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 22:52:24.757029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.757001    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/premium-w2-56f3fe-kserve-769d49fff-469xt"]
Oct 30 22:52:24.917437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.917343    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 22:52:24.920526 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.920500    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-56f3fe/distinct-w2-56f3fe-kserve-8575687f54-zrhpr"]
Oct 30 22:52:24.942503 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.942469    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" path="/var/lib/kubelet/pods/186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16/volumes"
Oct 30 22:52:24.943003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:24.942979    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" path="/var/lib/kubelet/pods/28c383d2-e556-40d6-a361-2291d3e2613d/volumes"
Oct 30 22:52:30.620530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:30.620495    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:52:33.852137 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:33.852099    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"]
Oct 30 22:52:33.902529 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:33.902493    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-4d14e0-zfmqv"]
Oct 30 22:52:34.942515 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:34.942465    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8170c4a-e7eb-4e0f-bda0-a2f367024172" path="/var/lib/kubelet/pods/d8170c4a-e7eb-4e0f-bda0-a2f367024172/volumes"
Oct 30 22:52:37.820777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.820732    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"]
Oct 30 22:52:37.821463 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821441    4882 cpu_manager.go:414] "containerMap: removing container" podUID="561e0967-db90-4d98-b691-86621581dbed" containerName="revoke-keys"
Oct 30 22:52:37.821560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821466    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="561e0967-db90-4d98-b691-86621581dbed" containerName="revoke-keys"
Oct 30 22:52:37.821560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821511    4882 cpu_manager.go:414] "containerMap: removing container" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main"
Oct 30 22:52:37.821560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821521    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main"
Oct 30 22:52:37.821560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821537    4882 cpu_manager.go:414] "containerMap: removing container" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main"
Oct 30 22:52:37.821560 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821544    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main"
Oct 30 22:52:37.821863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821680    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="28c383d2-e556-40d6-a361-2291d3e2613d" containerName="main"
Oct 30 22:52:37.821863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821696    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="186a9dc2-ff2f-47d4-bb0a-6f4678b7dc16" containerName="main"
Oct 30 22:52:37.821863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.821713    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="561e0967-db90-4d98-b691-86621581dbed" containerName="revoke-keys"
Oct 30 22:52:37.825116 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.825092    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.869969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.869936    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45nmk\" (UniqueName: \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.869969 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.869971    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.870218 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.870015    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.872496 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.872465    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"]
Oct 30 22:52:37.876476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.876455    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:37.877975 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.877952    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"]
Oct 30 22:52:37.920430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.920396    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:52:37.920642 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.920489    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:52:37.931007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.930975    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"]
Oct 30 22:52:37.971406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971373    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:37.971605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971432    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45nmk\" (UniqueName: \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.971605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971518    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.971605 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971595    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.971789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971628    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:37.971789 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.971723    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fd9ws\" (UniqueName: \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:37.972307 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.972288    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:37.974066 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:37.974049    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:38.033505 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.033471    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-45nmk\" (UniqueName: \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\") pod \"maas-api-revoke-sub-462f57d1-ptfkv\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:38.073106 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.073008    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.073303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.073143    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.073303 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.073193    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fd9ws\" (UniqueName: \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.073723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.073700    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.075625 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.075601    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.135293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.135258    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fd9ws\" (UniqueName: \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\") pod \"maas-api-revoke-sub-8977f241-4kpvm\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.137991 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.137971    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:38.190548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.190510    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:38.516348 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:52:38.516309    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod615a4250_0077_47bf_a165_1f30f28c34a8.slice/crio-9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7 WatchSource:0}: Error finding container 9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7: Status 404 returned error can't find the container with id 9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7
Oct 30 22:52:38.518322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.518289    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"]
Oct 30 22:52:38.573956 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:52:38.573914    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe6ab3ad_6064_4bfd_9613_9f004c771730.slice/crio-199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f WatchSource:0}: Error finding container 199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f: Status 404 returned error can't find the container with id 199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f
Oct 30 22:52:38.576826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.576713    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"]
Oct 30 22:52:38.685068 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.685032    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv" event={"ID":"615a4250-0077-47bf-a165-1f30f28c34a8","Type":"ContainerStarted","Data":"9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7"}
Oct 30 22:52:38.686292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:38.686265    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm" event={"ID":"be6ab3ad-6064-4bfd-9613-9f004c771730","Type":"ContainerStarted","Data":"199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f"}
Oct 30 22:52:39.694260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:39.694160    4882 generic.go:356] "Generic (PLEG): container finished" podID="615a4250-0077-47bf-a165-1f30f28c34a8" containerID="6a700509d9b1475bced261c66902a072b0a9e44b93437cb4a5b92f0cab05fc91" exitCode=0
Oct 30 22:52:39.694722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:39.694250    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv" event={"ID":"615a4250-0077-47bf-a165-1f30f28c34a8","Type":"ContainerDied","Data":"6a700509d9b1475bced261c66902a072b0a9e44b93437cb4a5b92f0cab05fc91"}
Oct 30 22:52:39.695906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:39.695880    4882 generic.go:356] "Generic (PLEG): container finished" podID="be6ab3ad-6064-4bfd-9613-9f004c771730" containerID="7e4f21c1cda7aaaa69f374ed4c3320e79d811faefdf6acd475dba056cdd339fa" exitCode=0
Oct 30 22:52:39.696012 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:39.695968    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm" event={"ID":"be6ab3ad-6064-4bfd-9613-9f004c771730","Type":"ContainerDied","Data":"7e4f21c1cda7aaaa69f374ed4c3320e79d811faefdf6acd475dba056cdd339fa"}
Oct 30 22:52:40.870371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.870343    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:40.874016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.873994    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:40.900609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900571    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\") pod \"be6ab3ad-6064-4bfd-9613-9f004c771730\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") "
Oct 30 22:52:40.900791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900715    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\" (UniqueName: \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\") pod \"615a4250-0077-47bf-a165-1f30f28c34a8\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") "
Oct 30 22:52:40.900791 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900773    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\") pod \"be6ab3ad-6064-4bfd-9613-9f004c771730\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") "
Oct 30 22:52:40.900931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900804    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\" (UniqueName: \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\") pod \"be6ab3ad-6064-4bfd-9613-9f004c771730\" (UID: \"be6ab3ad-6064-4bfd-9613-9f004c771730\") "
Oct 30 22:52:40.900931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900845    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\") pod \"615a4250-0077-47bf-a165-1f30f28c34a8\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") "
Oct 30 22:52:40.900931 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.900882    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\") pod \"615a4250-0077-47bf-a165-1f30f28c34a8\" (UID: \"615a4250-0077-47bf-a165-1f30f28c34a8\") "
Oct 30 22:52:40.901167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.901140    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca" pod "be6ab3ad-6064-4bfd-9613-9f004c771730" (UID: "be6ab3ad-6064-4bfd-9613-9f004c771730"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:40.901335 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.901317    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:40.901454 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.901336    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca" pod "615a4250-0077-47bf-a165-1f30f28c34a8" (UID: "615a4250-0077-47bf-a165-1f30f28c34a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:52:40.902908 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.902882    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca" pod "be6ab3ad-6064-4bfd-9613-9f004c771730" (UID: "be6ab3ad-6064-4bfd-9613-9f004c771730"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:40.903025 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.903007    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk" pod "615a4250-0077-47bf-a165-1f30f28c34a8" (UID: "615a4250-0077-47bf-a165-1f30f28c34a8"). InnerVolumeSpecName "kube-api-access-45nmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:40.903373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.903348    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws" pod "be6ab3ad-6064-4bfd-9613-9f004c771730" (UID: "be6ab3ad-6064-4bfd-9613-9f004c771730"). InnerVolumeSpecName "kube-api-access-fd9ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:52:40.903373 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:40.903359    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca" pod "615a4250-0077-47bf-a165-1f30f28c34a8" (UID: "615a4250-0077-47bf-a165-1f30f28c34a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:52:41.002021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.001981    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be6ab3ad-6064-4bfd-9613-9f004c771730-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:41.002021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.002011    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45nmk\" (UniqueName: \"kubernetes.io/projected/615a4250-0077-47bf-a165-1f30f28c34a8-kube-api-access-45nmk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:41.002021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.002021    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fd9ws\" (UniqueName: \"kubernetes.io/projected/be6ab3ad-6064-4bfd-9613-9f004c771730-kube-api-access-fd9ws\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:41.002021 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.002030    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:41.002413 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.002039    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/615a4250-0077-47bf-a165-1f30f28c34a8-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:52:41.710101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.710067    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"
Oct 30 22:52:41.710101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.710085    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv" event={"ID":"615a4250-0077-47bf-a165-1f30f28c34a8","Type":"ContainerDied","Data":"9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7"}
Oct 30 22:52:41.710346 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.710138    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9fc952281743e9d4535bbe6fc0a8f554a3c2b18e4009c23501c19d0953d15fc7"
Oct 30 22:52:41.711547 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.711526    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"
Oct 30 22:52:41.711746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.711557    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm" event={"ID":"be6ab3ad-6064-4bfd-9613-9f004c771730","Type":"ContainerDied","Data":"199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f"}
Oct 30 22:52:41.711746 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:41.711617    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="199e2f9ec4f961a2ec39974ac8f22a85e4645a082996a2552c9869d7be58e66f"
Oct 30 22:52:50.807662 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:52:50.807616    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:52:57.633230 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:57.633203    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:52:57.651737 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:52:57.651706    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:53:00.625287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:00.625250    4882 scope.go:122] "RemoveContainer" containerID="cc36d87e45c24afe4caa4aa115953ff14c95cca40b9e8bb49493477787c1a555"
Oct 30 22:53:00.635476 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:00.635451    4882 scope.go:122] "RemoveContainer" containerID="af7406cd2092a5c6a6d48b819db1efe78e43b5e099eb04fea9ea7a935174d796"
Oct 30 22:53:00.646667 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:00.646643    4882 scope.go:122] "RemoveContainer" containerID="3f54677d1c537a31f0acca584139cd8a72af58a12954c517d014c286706ddf8c"
Oct 30 22:53:06.899465 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:06.899385    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:53:12.219976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:12.219939    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"]
Oct 30 22:53:12.270014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:12.269983    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-56f3fe-xzvfc"]
Oct 30 22:53:12.941870 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:12.941825    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="561e0967-db90-4d98-b691-86621581dbed" path="/var/lib/kubelet/pods/561e0967-db90-4d98-b691-86621581dbed/volumes"
Oct 30 22:53:20.806337 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:53:20.806294    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:53:25.040357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040324    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"]
Oct 30 22:53:25.040875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040860    4882 cpu_manager.go:414] "containerMap: removing container" podUID="be6ab3ad-6064-4bfd-9613-9f004c771730" containerName="revoke-keys"
Oct 30 22:53:25.040924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040877    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be6ab3ad-6064-4bfd-9613-9f004c771730" containerName="revoke-keys"
Oct 30 22:53:25.040924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040895    4882 cpu_manager.go:414] "containerMap: removing container" podUID="615a4250-0077-47bf-a165-1f30f28c34a8" containerName="revoke-keys"
Oct 30 22:53:25.040924 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040903    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="615a4250-0077-47bf-a165-1f30f28c34a8" containerName="revoke-keys"
Oct 30 22:53:25.041032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040990    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="be6ab3ad-6064-4bfd-9613-9f004c771730" containerName="revoke-keys"
Oct 30 22:53:25.041032 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.040999    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="615a4250-0077-47bf-a165-1f30f28c34a8" containerName="revoke-keys"
Oct 30 22:53:25.045442 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.045417    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.098334 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.098299    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"]
Oct 30 22:53:25.145514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.145478    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:25.147357 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.147323    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:53:25.153486 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.153457    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.153607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.153514    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.153715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.153695    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rr4sl\" (UniqueName: \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.254416 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.254381    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rr4sl\" (UniqueName: \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.254632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.254444    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.254632 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.254493    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.255325 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.255286    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.257002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.256978    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.316886 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.316805    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rr4sl\" (UniqueName: \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\") pod \"maas-api-revoke-sub-d8ab1463-xffmg\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.358879 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.358838    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:25.730874 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:53:25.730844    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod77cb3f32_d9d4_4907_8795_9b6a83114aee.slice/crio-32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48 WatchSource:0}: Error finding container 32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48: Status 404 returned error can't find the container with id 32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48
Oct 30 22:53:25.733623 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.733558    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"]
Oct 30 22:53:25.988707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:25.988612    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg" event={"ID":"77cb3f32-d9d4-4907-8795-9b6a83114aee","Type":"ContainerStarted","Data":"32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48"}
Oct 30 22:53:26.996941 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:26.996859    4882 generic.go:356] "Generic (PLEG): container finished" podID="77cb3f32-d9d4-4907-8795-9b6a83114aee" containerID="26ecbe5d9e874d8d50510421f7d13f873c79bbd47819f6411a9a667a70ddc99f" exitCode=0
Oct 30 22:53:26.997317 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:26.996941    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg" event={"ID":"77cb3f32-d9d4-4907-8795-9b6a83114aee","Type":"ContainerDied","Data":"26ecbe5d9e874d8d50510421f7d13f873c79bbd47819f6411a9a667a70ddc99f"}
Oct 30 22:53:28.142610 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.142565    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:28.286612 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.286515    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\") pod \"77cb3f32-d9d4-4907-8795-9b6a83114aee\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") "
Oct 30 22:53:28.286776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.286658    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\" (UniqueName: \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\") pod \"77cb3f32-d9d4-4907-8795-9b6a83114aee\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") "
Oct 30 22:53:28.286776 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.286764    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\") pod \"77cb3f32-d9d4-4907-8795-9b6a83114aee\" (UID: \"77cb3f32-d9d4-4907-8795-9b6a83114aee\") "
Oct 30 22:53:28.287149 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.287117    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca" pod "77cb3f32-d9d4-4907-8795-9b6a83114aee" (UID: "77cb3f32-d9d4-4907-8795-9b6a83114aee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:28.288680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.288648    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca" pod "77cb3f32-d9d4-4907-8795-9b6a83114aee" (UID: "77cb3f32-d9d4-4907-8795-9b6a83114aee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:28.289199 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.289179    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl" pod "77cb3f32-d9d4-4907-8795-9b6a83114aee" (UID: "77cb3f32-d9d4-4907-8795-9b6a83114aee"). InnerVolumeSpecName "kube-api-access-rr4sl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:28.387855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.387812    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.387855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.387845    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77cb3f32-d9d4-4907-8795-9b6a83114aee-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.387855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.387857    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rr4sl\" (UniqueName: \"kubernetes.io/projected/77cb3f32-d9d4-4907-8795-9b6a83114aee-kube-api-access-rr4sl\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:28.451097 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:28.451062    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:53:29.010742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:29.010707    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg"
Oct 30 22:53:29.010926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:29.010742    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8ab1463-xffmg" event={"ID":"77cb3f32-d9d4-4907-8795-9b6a83114aee","Type":"ContainerDied","Data":"32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48"}
Oct 30 22:53:29.010926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:29.010791    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32c59086d2dfbc7c724ed3e8bb619d5985d28cd609e80f6cbb142fdaada52c48"
Oct 30 22:53:42.051578 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:42.051538    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"]
Oct 30 22:53:42.101850 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:42.101804    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a6ad0560-tvckc"]
Oct 30 22:53:42.942548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:42.942511    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6" path="/var/lib/kubelet/pods/dcb46564-1dff-4f59-9ad1-8ff2e09fb0e6/volumes"
Oct 30 22:53:43.051268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:43.051232    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"]
Oct 30 22:53:43.101745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:43.101697    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-459ad77a-kfnrk"]
Oct 30 22:53:43.245133 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:43.245048    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:53:44.943121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:44.943077    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9b140d6-8a80-4af2-815e-da7355134862" path="/var/lib/kubelet/pods/b9b140d6-8a80-4af2-815e-da7355134862/volumes"
Oct 30 22:53:50.812604 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:53:50.812551    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:53:51.647265 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.647230    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"]
Oct 30 22:53:51.647803 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.647786    4882 cpu_manager.go:414] "containerMap: removing container" podUID="77cb3f32-d9d4-4907-8795-9b6a83114aee" containerName="revoke-keys"
Oct 30 22:53:51.647860 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.647804    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77cb3f32-d9d4-4907-8795-9b6a83114aee" containerName="revoke-keys"
Oct 30 22:53:51.647906 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.647896    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="77cb3f32-d9d4-4907-8795-9b6a83114aee" containerName="revoke-keys"
Oct 30 22:53:51.651258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.651242    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.706227 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.706195    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"]
Oct 30 22:53:51.726060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.726016    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kvh77\" (UniqueName: \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.726060 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.726070    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.726282 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.726220    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.752691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.752634    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:53:51.752863 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.752698    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:53:51.827837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.827788    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kvh77\" (UniqueName: \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.827837 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.827845    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.828285 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.827900    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.828536 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.828498    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.830292 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.830263    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.889552 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.889510    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kvh77\" (UniqueName: \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\") pod \"maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:51.964497 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:51.964417    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:52.334181 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:53:52.334143    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2891047e_8ee3_4807_9902_1a5ed8c8e695.slice/crio-3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad WatchSource:0}: Error finding container 3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad: Status 404 returned error can't find the container with id 3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad
Oct 30 22:53:52.336541 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:52.336516    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"]
Oct 30 22:53:53.162741 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:53.162701    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc" event={"ID":"2891047e-8ee3-4807-9902-1a5ed8c8e695","Type":"ContainerStarted","Data":"78afab332aa45fbaf4bb15d26fd1c9232c9ea9b98769d5878d0397423aa57c2d"}
Oct 30 22:53:53.163272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:53.162753    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc" event={"ID":"2891047e-8ee3-4807-9902-1a5ed8c8e695","Type":"ContainerStarted","Data":"3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad"}
Oct 30 22:53:53.283559 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:53.283500    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc" podStartSLOduration=2.283486399 podStartE2EDuration="2.283486399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:53:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:53:53.280823173 +0000 UTC m=+4556.454147600" watchObservedRunningTime="2026-10-30 22:53:53.283486399 +0000 UTC m=+4556.456810825"
Oct 30 22:53:54.170343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:54.170304    4882 generic.go:356] "Generic (PLEG): container finished" podID="2891047e-8ee3-4807-9902-1a5ed8c8e695" containerID="78afab332aa45fbaf4bb15d26fd1c9232c9ea9b98769d5878d0397423aa57c2d" exitCode=0
Oct 30 22:53:54.170785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:54.170374    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc" event={"ID":"2891047e-8ee3-4807-9902-1a5ed8c8e695","Type":"ContainerDied","Data":"78afab332aa45fbaf4bb15d26fd1c9232c9ea9b98769d5878d0397423aa57c2d"}
Oct 30 22:53:55.319039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.319010    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:55.363781 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.363742    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\" (UniqueName: \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\") pod \"2891047e-8ee3-4807-9902-1a5ed8c8e695\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") "
Oct 30 22:53:55.363993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.363807    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\") pod \"2891047e-8ee3-4807-9902-1a5ed8c8e695\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") "
Oct 30 22:53:55.363993 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.363856    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\") pod \"2891047e-8ee3-4807-9902-1a5ed8c8e695\" (UID: \"2891047e-8ee3-4807-9902-1a5ed8c8e695\") "
Oct 30 22:53:55.364278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.364248    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca" pod "2891047e-8ee3-4807-9902-1a5ed8c8e695" (UID: "2891047e-8ee3-4807-9902-1a5ed8c8e695"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:53:55.366000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.365976    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77" pod "2891047e-8ee3-4807-9902-1a5ed8c8e695" (UID: "2891047e-8ee3-4807-9902-1a5ed8c8e695"). InnerVolumeSpecName "kube-api-access-kvh77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:53:55.366105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.365998    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca" pod "2891047e-8ee3-4807-9902-1a5ed8c8e695" (UID: "2891047e-8ee3-4807-9902-1a5ed8c8e695"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:53:55.465680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.465581    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:55.465680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.465640    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2891047e-8ee3-4807-9902-1a5ed8c8e695-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:55.465680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:55.465657    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kvh77\" (UniqueName: \"kubernetes.io/projected/2891047e-8ee3-4807-9902-1a5ed8c8e695-kube-api-access-kvh77\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:53:56.184829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:56.184801    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"
Oct 30 22:53:56.185051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:56.184838    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc" event={"ID":"2891047e-8ee3-4807-9902-1a5ed8c8e695","Type":"ContainerDied","Data":"3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad"}
Oct 30 22:53:56.185051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:53:56.184886    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3009d10e81b4c06a6f2a2796aa3cc6f95c45aa52bfce7e008d66db8f3e6616ad"
Oct 30 22:54:00.697238 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:00.697206    4882 scope.go:122] "RemoveContainer" containerID="89da38e0c1f4a29f283e15b30067d6263013e2a54d7587cdb540e13bec3ef98a"
Oct 30 22:54:00.706853 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:00.706833    4882 scope.go:122] "RemoveContainer" containerID="4fc253a7632fd77cc5103f119901a7fd35279ca18c1432c66c05e94b9fceebed"
Oct 30 22:54:00.716077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:00.716056    4882 scope.go:122] "RemoveContainer" containerID="a923d6344c5b791b57fc0d0b895dddcf0637d8227c95e3888fb0d6f9463dddb5"
Oct 30 22:54:00.725063 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:00.725035    4882 scope.go:122] "RemoveContainer" containerID="bdd083731890629e386a3dd0f65d146622edb2e735429fb5b9d3dec5d6f4c836"
Oct 30 22:54:03.172921 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:03.172888    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:54:03.278884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:03.278845    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:54:08.173872 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.173808    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" podUID="d87fc4f9-8c7b-424b-b31a-67935563bec0" containerName="payload-processing" containerID="cri-o://edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2" gracePeriod=25
Oct 30 22:54:08.280141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.280081    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" podUID="42b6051e-995a-460c-b8f6-07592f9d6846" containerName="payload-pre-processing" containerID="cri-o://efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1" gracePeriod=25
Oct 30 22:54:08.445868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.445836    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:54:08.545972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.545942    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:54:08.602905 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.602872    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\") pod \"d87fc4f9-8c7b-424b-b31a-67935563bec0\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") "
Oct 30 22:54:08.603120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.602986    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\" (UniqueName: \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\") pod \"d87fc4f9-8c7b-424b-b31a-67935563bec0\" (UID: \"d87fc4f9-8c7b-424b-b31a-67935563bec0\") "
Oct 30 22:54:08.603278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.603253    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume" pod "d87fc4f9-8c7b-424b-b31a-67935563bec0" (UID: "d87fc4f9-8c7b-424b-b31a-67935563bec0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:08.605195 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.605174    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d" pod "d87fc4f9-8c7b-424b-b31a-67935563bec0" (UID: "d87fc4f9-8c7b-424b-b31a-67935563bec0"). InnerVolumeSpecName "kube-api-access-mw84d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:08.704269 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.704172    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\" (UniqueName: \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\") pod \"42b6051e-995a-460c-b8f6-07592f9d6846\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") "
Oct 30 22:54:08.704451 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.704342    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\") pod \"42b6051e-995a-460c-b8f6-07592f9d6846\" (UID: \"42b6051e-995a-460c-b8f6-07592f9d6846\") "
Oct 30 22:54:08.704770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.704737    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d87fc4f9-8c7b-424b-b31a-67935563bec0-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:08.704770 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.704768    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mw84d\" (UniqueName: \"kubernetes.io/projected/d87fc4f9-8c7b-424b-b31a-67935563bec0-kube-api-access-mw84d\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:08.704955 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.704794    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume" pod "42b6051e-995a-460c-b8f6-07592f9d6846" (UID: "42b6051e-995a-460c-b8f6-07592f9d6846"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:08.706551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.706527    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs" pod "42b6051e-995a-460c-b8f6-07592f9d6846" (UID: "42b6051e-995a-460c-b8f6-07592f9d6846"). InnerVolumeSpecName "kube-api-access-2vqhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:08.805387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.805337    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vqhs\" (UniqueName: \"kubernetes.io/projected/42b6051e-995a-460c-b8f6-07592f9d6846-kube-api-access-2vqhs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:08.805387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:08.805377    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/42b6051e-995a-460c-b8f6-07592f9d6846-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:09.266176 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.266143    4882 generic.go:356] "Generic (PLEG): container finished" podID="d87fc4f9-8c7b-424b-b31a-67935563bec0" containerID="edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2" exitCode=0
Oct 30 22:54:09.266696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.266190    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"
Oct 30 22:54:09.266696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.266230    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" event={"ID":"d87fc4f9-8c7b-424b-b31a-67935563bec0","Type":"ContainerDied","Data":"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"}
Oct 30 22:54:09.266696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.266295    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k" event={"ID":"d87fc4f9-8c7b-424b-b31a-67935563bec0","Type":"ContainerDied","Data":"a22c781fab4c5bb0490d3012c60304e7a77d0890a2d855a24668f71d84a90df2"}
Oct 30 22:54:09.266696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.266314    4882 scope.go:122] "RemoveContainer" containerID="edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"
Oct 30 22:54:09.267830 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.267805    4882 generic.go:356] "Generic (PLEG): container finished" podID="42b6051e-995a-460c-b8f6-07592f9d6846" containerID="efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1" exitCode=0
Oct 30 22:54:09.267926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.267857    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"
Oct 30 22:54:09.267926 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.267887    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" event={"ID":"42b6051e-995a-460c-b8f6-07592f9d6846","Type":"ContainerDied","Data":"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"}
Oct 30 22:54:09.268051 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.267940    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7" event={"ID":"42b6051e-995a-460c-b8f6-07592f9d6846","Type":"ContainerDied","Data":"b99f07bc5facc71bbc92d9a9530d5918fa98b1a1df8ed7e65d1e37b8ef0cc0ce"}
Oct 30 22:54:09.276374 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.276354    4882 scope.go:122] "RemoveContainer" containerID="edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"
Oct 30 22:54:09.276678 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:09.276658    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2\": container with ID starting with edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2 not found: ID does not exist" containerID="edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"
Oct 30 22:54:09.276742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.276688    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2"} err="failed to get container status \"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2\": rpc error: code = NotFound desc = could not find container \"edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2\": container with ID starting with edf14d9537d799518e80716a2df582a88eec8e140b7b177179b910664b555dc2 not found: ID does not exist"
Oct 30 22:54:09.276742 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.276706    4882 scope.go:122] "RemoveContainer" containerID="efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"
Oct 30 22:54:09.286131 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.286105    4882 scope.go:122] "RemoveContainer" containerID="efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"
Oct 30 22:54:09.286422 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:09.286397    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1\": container with ID starting with efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1 not found: ID does not exist" containerID="efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"
Oct 30 22:54:09.286533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.286428    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1"} err="failed to get container status \"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1\": rpc error: code = NotFound desc = could not find container \"efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1\": container with ID starting with efea71741dc50902f95b40d927c89204e79c57f3ecee36b8c901a5e626ed5df1 not found: ID does not exist"
Oct 30 22:54:09.433000 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.432962    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:54:09.483709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.483616    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-extmod-a-0c4786-5d5b944779-r8f5k"]
Oct 30 22:54:09.596036 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.596002    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:54:09.603888 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:09.603859    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-a-0c4786-5794759b99-6k4f7"]
Oct 30 22:54:10.941691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:10.941656    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42b6051e-995a-460c-b8f6-07592f9d6846" path="/var/lib/kubelet/pods/42b6051e-995a-460c-b8f6-07592f9d6846/volumes"
Oct 30 22:54:10.942120 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:10.942068    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d87fc4f9-8c7b-424b-b31a-67935563bec0" path="/var/lib/kubelet/pods/d87fc4f9-8c7b-424b-b31a-67935563bec0/volumes"
Oct 30 22:54:18.860708 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:18.860670    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"]
Oct 30 22:54:18.911491 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:18.911458    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-a-0c4786-kbnkc"]
Oct 30 22:54:18.942757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:18.942720    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2891047e-8ee3-4807-9902-1a5ed8c8e695" path="/var/lib/kubelet/pods/2891047e-8ee3-4807-9902-1a5ed8c8e695/volumes"
Oct 30 22:54:20.806277 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:20.806236    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:54:22.801169 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:22.801125    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:54:26.059462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.059420    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:54:26.059895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.059657    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy" containerID="cri-o://e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166" gracePeriod=30
Oct 30 22:54:26.931104 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931056    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"]
Oct 30 22:54:26.931537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931524    4882 cpu_manager.go:414] "containerMap: removing container" podUID="2891047e-8ee3-4807-9902-1a5ed8c8e695" containerName="revoke-keys"
Oct 30 22:54:26.931607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931538    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2891047e-8ee3-4807-9902-1a5ed8c8e695" containerName="revoke-keys"
Oct 30 22:54:26.931607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931552    4882 cpu_manager.go:414] "containerMap: removing container" podUID="d87fc4f9-8c7b-424b-b31a-67935563bec0" containerName="payload-processing"
Oct 30 22:54:26.931607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931557    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d87fc4f9-8c7b-424b-b31a-67935563bec0" containerName="payload-processing"
Oct 30 22:54:26.931719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931610    4882 cpu_manager.go:414] "containerMap: removing container" podUID="42b6051e-995a-460c-b8f6-07592f9d6846" containerName="payload-pre-processing"
Oct 30 22:54:26.931719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931617    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42b6051e-995a-460c-b8f6-07592f9d6846" containerName="payload-pre-processing"
Oct 30 22:54:26.931719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931683    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="42b6051e-995a-460c-b8f6-07592f9d6846" containerName="payload-pre-processing"
Oct 30 22:54:26.931719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931694    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="2891047e-8ee3-4807-9902-1a5ed8c8e695" containerName="revoke-keys"
Oct 30 22:54:26.931719 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.931700    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="d87fc4f9-8c7b-424b-b31a-67935563bec0" containerName="payload-processing"
Oct 30 22:54:26.938468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.938445    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:26.991884 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.991851    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"]
Oct 30 22:54:26.992448 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.992421    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:26.992607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.992477    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:26.992607 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:26.992511    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmfvk\" (UniqueName: \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.038544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.038509    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:27.038754 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.038686    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:54:27.093291 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.093257    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.093809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.093306    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.093809 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.093425    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmfvk\" (UniqueName: \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.094001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.093982    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.095734 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.095712    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.156816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.156780    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmfvk\" (UniqueName: \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\") pod \"maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.252371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.252287    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:27.624533 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:54:27.624491    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35208c37_8d16_4857_9e9d_90789877991e.slice/crio-b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9 WatchSource:0}: Error finding container b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9: Status 404 returned error can't find the container with id b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9
Oct 30 22:54:27.626430 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:27.626407    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"]
Oct 30 22:54:28.386077 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:28.386033    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8" event={"ID":"35208c37-8d16-4857-9e9d-90789877991e","Type":"ContainerStarted","Data":"cef918b2a799440098a395ad83e144a85eb6a6675335dd5682f609bc64ad0a10"}
Oct 30 22:54:28.386495 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:28.386100    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8" event={"ID":"35208c37-8d16-4857-9e9d-90789877991e","Type":"ContainerStarted","Data":"b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9"}
Oct 30 22:54:28.514322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:28.514262    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8" podStartSLOduration=2.514246332 podStartE2EDuration="2.514246332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:28.511759883 +0000 UTC m=+4591.685084310" watchObservedRunningTime="2026-10-30 22:54:28.514246332 +0000 UTC m=+4591.687570758"
Oct 30 22:54:29.393835 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:29.393794    4882 generic.go:356] "Generic (PLEG): container finished" podID="35208c37-8d16-4857-9e9d-90789877991e" containerID="cef918b2a799440098a395ad83e144a85eb6a6675335dd5682f609bc64ad0a10" exitCode=0
Oct 30 22:54:29.394405 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:29.393885    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8" event={"ID":"35208c37-8d16-4857-9e9d-90789877991e","Type":"ContainerDied","Data":"cef918b2a799440098a395ad83e144a85eb6a6675335dd5682f609bc64ad0a10"}
Oct 30 22:54:30.540744 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.540720    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:30.629427 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.629394    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\") pod \"35208c37-8d16-4857-9e9d-90789877991e\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") "
Oct 30 22:54:30.629631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.629472    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\" (UniqueName: \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\") pod \"35208c37-8d16-4857-9e9d-90789877991e\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") "
Oct 30 22:54:30.629631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.629520    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\") pod \"35208c37-8d16-4857-9e9d-90789877991e\" (UID: \"35208c37-8d16-4857-9e9d-90789877991e\") "
Oct 30 22:54:30.629967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.629939    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca" pod "35208c37-8d16-4857-9e9d-90789877991e" (UID: "35208c37-8d16-4857-9e9d-90789877991e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:30.631631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.631606    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca" pod "35208c37-8d16-4857-9e9d-90789877991e" (UID: "35208c37-8d16-4857-9e9d-90789877991e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:30.631723 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.631609    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk" pod "35208c37-8d16-4857-9e9d-90789877991e" (UID: "35208c37-8d16-4857-9e9d-90789877991e"). InnerVolumeSpecName "kube-api-access-wmfvk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:30.731349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.731255    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/35208c37-8d16-4857-9e9d-90789877991e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:30.731349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.731295    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmfvk\" (UniqueName: \"kubernetes.io/projected/35208c37-8d16-4857-9e9d-90789877991e-kube-api-access-wmfvk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:30.731349 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:30.731307    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/35208c37-8d16-4857-9e9d-90789877991e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.302027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.302000    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:54:31.337121 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337094    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337142    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337165    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337212    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337235    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337296 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337266    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337342    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337395    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337554 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337435    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\") pod \"4f04c653-84be-4ae7-94c7-99a00323d294\" (UID: \"4f04c653-84be-4ae7-94c7-99a00323d294\") "
Oct 30 22:54:31.337759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337543    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.337759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337660    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.337759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337543    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:31.337759 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337683    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.337972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337767    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.337972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337911    4882 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.337972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337934    4882 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-credential-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.337972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337949    4882 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f04c653-84be-4ae7-94c7-99a00323d294-istiod-ca-cert\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.337972 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337962    4882 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-data\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.338260 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.337978    4882 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-workload-socket\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.339637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.339607    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.339637 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.339623    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:54:31.339796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.339675    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Oct 30 22:54:31.339796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.339755    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh" pod "4f04c653-84be-4ae7-94c7-99a00323d294" (UID: "4f04c653-84be-4ae7-94c7-99a00323d294"). InnerVolumeSpecName "kube-api-access-6lwkh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:31.407910 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.407882    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"
Oct 30 22:54:31.408101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.407924    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8" event={"ID":"35208c37-8d16-4857-9e9d-90789877991e","Type":"ContainerDied","Data":"b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9"}
Oct 30 22:54:31.408101 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.407971    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7f8dab152dc1b3a2ff04bdc31ea79312d3f39abb4365617b5fe2800afa4d2b9"
Oct 30 22:54:31.409966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.409865    4882 generic.go:356] "Generic (PLEG): container finished" podID="4f04c653-84be-4ae7-94c7-99a00323d294" containerID="e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166" exitCode=0
Oct 30 22:54:31.409966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.409925    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerDied","Data":"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"}
Oct 30 22:54:31.409966 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.409941    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"
Oct 30 22:54:31.410246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.409987    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc" event={"ID":"4f04c653-84be-4ae7-94c7-99a00323d294","Type":"ContainerDied","Data":"bf5eed93ec7b75aad6cea329dddad7759c7c68f4edb4ac98583880c6b2fa0861"}
Oct 30 22:54:31.410246 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.410021    4882 scope.go:122] "RemoveContainer" containerID="e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"
Oct 30 22:54:31.420217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.420197    4882 scope.go:122] "RemoveContainer" containerID="9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"
Oct 30 22:54:31.429566 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.429545    4882 scope.go:122] "RemoveContainer" containerID="e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"
Oct 30 22:54:31.429877 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:31.429838    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166\": container with ID starting with e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166 not found: ID does not exist" containerID="e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"
Oct 30 22:54:31.429965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.429886    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166"} err="failed to get container status \"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166\": rpc error: code = NotFound desc = could not find container \"e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166\": container with ID starting with e35de2525c05f41dd8971f4557366573e92478158ab943e7f3fb66ce6f50e166 not found: ID does not exist"
Oct 30 22:54:31.429965 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.429904    4882 scope.go:122] "RemoveContainer" containerID="9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"
Oct 30 22:54:31.430144 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:31.430126    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789\": container with ID starting with 9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789 not found: ID does not exist" containerID="9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"
Oct 30 22:54:31.430188 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.430150    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789"} err="failed to get container status \"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789\": rpc error: code = NotFound desc = could not find container \"9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789\": container with ID starting with 9b3ce26963e9fb06c344bf6a2cd67cf564a9d47149d4f999cc008d51e54ee789 not found: ID does not exist"
Oct 30 22:54:31.438709 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.438690    4882 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f04c653-84be-4ae7-94c7-99a00323d294-istio-podinfo\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.438782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.438712    4882 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-istio-token\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.438782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.438725    4882 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f04c653-84be-4ae7-94c7-99a00323d294-istio-envoy\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.438782 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.438735    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lwkh\" (UniqueName: \"kubernetes.io/projected/4f04c653-84be-4ae7-94c7-99a00323d294-kube-api-access-6lwkh\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:31.572609 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.572556    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:54:31.576868 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:31.576843    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-extmod-a-0c4786-openshift-default-fdc958bcc-4p4qc"]
Oct 30 22:54:32.943426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:32.943387    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" path="/var/lib/kubelet/pods/4f04c653-84be-4ae7-94c7-99a00323d294/volumes"
Oct 30 22:54:38.579656 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:38.579555    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:54:43.580896 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:43.580834    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing" containerID="cri-o://44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37" gracePeriod=25
Oct 30 22:54:43.771010 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:43.770962    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"]
Oct 30 22:54:43.774530 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:43.774498    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:54:43.821368 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:43.821336    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-extmod-b-9af9e7-j97w8"]
Oct 30 22:54:44.942033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:44.942000    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35208c37-8d16-4857-9e9d-90789877991e" path="/var/lib/kubelet/pods/35208c37-8d16-4857-9e9d-90789877991e/volumes"
Oct 30 22:54:47.446514 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:47.446480    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:54:50.807812 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:54:50.807772    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:54:52.081728 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.081696    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"]
Oct 30 22:54:52.082172 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082158    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:54:52.082219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082175    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:54:52.082219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082213    4882 cpu_manager.go:414] "containerMap: removing container" podUID="35208c37-8d16-4857-9e9d-90789877991e" containerName="revoke-keys"
Oct 30 22:54:52.082219 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082218    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35208c37-8d16-4857-9e9d-90789877991e" containerName="revoke-keys"
Oct 30 22:54:52.082322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082293    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="35208c37-8d16-4857-9e9d-90789877991e" containerName="revoke-keys"
Oct 30 22:54:52.082322 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.082303    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:54:52.085813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.085795    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.140281 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.140251    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:54:52.140981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.140958    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"]
Oct 30 22:54:52.187936 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.187906    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:54:52.251996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.251962    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.251996 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.252003    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.252240 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.252141    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wnzk\" (UniqueName: \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.353546 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.353442    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wnzk\" (UniqueName: \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.353796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.353568    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.353796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.353632    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.354293 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.354267    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.356258 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.356236    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.420550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.420511    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wnzk\" (UniqueName: \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:52.699014 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:52.698922    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:53.076040 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:54:53.075991    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ca8f7b5_09cd_4b38_8548_c0738a5b3d1f.slice/crio-81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1 WatchSource:0}: Error finding container 81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1: Status 404 returned error can't find the container with id 81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1
Oct 30 22:54:53.077550 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:53.077525    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"]
Oct 30 22:54:53.539361 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:53.539311    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7" event={"ID":"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f","Type":"ContainerStarted","Data":"f1b25a083d8d3779324d7f1a237e656d9512dac75145558754c30a38bb10a6b7"}
Oct 30 22:54:53.539792 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:53.539377    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7" event={"ID":"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f","Type":"ContainerStarted","Data":"81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1"}
Oct 30 22:54:53.656691 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:53.656566    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7" podStartSLOduration=1.656553202 podStartE2EDuration="1.656553202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 22:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:54:53.655494689 +0000 UTC m=+4616.828819115" watchObservedRunningTime="2026-10-30 22:54:53.656553202 +0000 UTC m=+4616.829877628"
Oct 30 22:54:53.774680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:53.774640    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:54:54.547469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:54.547434    4882 generic.go:356] "Generic (PLEG): container finished" podID="1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" containerID="f1b25a083d8d3779324d7f1a237e656d9512dac75145558754c30a38bb10a6b7" exitCode=0
Oct 30 22:54:54.547940 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:54.547524    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7" event={"ID":"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f","Type":"ContainerDied","Data":"f1b25a083d8d3779324d7f1a237e656d9512dac75145558754c30a38bb10a6b7"}
Oct 30 22:54:55.691371 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.691342    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:55.786171 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.786137    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\") pod \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") "
Oct 30 22:54:55.786367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.786257    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\" (UniqueName: \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\") pod \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") "
Oct 30 22:54:55.786367 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.786306    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\") pod \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\" (UID: \"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f\") "
Oct 30 22:54:55.786703 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.786678    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca" pod "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" (UID: "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:54:55.788537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.788508    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk" pod "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" (UID: "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f"). InnerVolumeSpecName "kube-api-access-8wnzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:54:55.788715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.788514    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca" pod "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" (UID: "1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:54:55.887456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.887419    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:55.887456 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.887456    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wnzk\" (UniqueName: \"kubernetes.io/projected/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-kube-api-access-8wnzk\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:55.887715 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:55.887471    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:54:56.559875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:56.559844    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"
Oct 30 22:54:56.559875 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:56.559851    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7" event={"ID":"1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f","Type":"ContainerDied","Data":"81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1"}
Oct 30 22:54:56.560105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:54:56.559898    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="81f84d75fdcc8500b06224594556c3a4bd42bfce7330eb5546274f1b3b00f0c1"
Oct 30 22:55:00.764971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:00.764935    4882 scope.go:122] "RemoveContainer" containerID="9498325fdfbda0b7b0101f47dccf265e1ff94e2f8c55d3eb57a0dcee5ea67918"
Oct 30 22:55:00.774494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:00.774469    4882 scope.go:122] "RemoveContainer" containerID="f60f513e5c6f121fd0c585ed96781df06572ea763bd670583e17c8b6b1528360"
Oct 30 22:55:00.783167 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:00.783140    4882 scope.go:122] "RemoveContainer" containerID="8cab88ec6786f370375145f29438cd48976d61f6d01c2d9bb233c3913dbd201d"
Oct 30 22:55:00.792125 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:00.792104    4882 scope.go:122] "RemoveContainer" containerID="8cc1c3776fd43acf51882fe8ab64a2fce9b119ea5bfb1e59b52b7699842e3052"
Oct 30 22:55:00.800640 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:00.800621    4882 scope.go:122] "RemoveContainer" containerID="4ff9e876ba4d218fc63096fe0cf0ab4d7404cb822d3131a1c65520af2b0980da"
Oct 30 22:55:03.686469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.686429    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:55:03.841657 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.841598    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:55:03.862544 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.862510    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\" (UniqueName: \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\") pod \"5e254661-d49d-4325-96f5-cde04db4358b\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") "
Oct 30 22:55:03.862761 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.862564    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\") pod \"5e254661-d49d-4325-96f5-cde04db4358b\" (UID: \"5e254661-d49d-4325-96f5-cde04db4358b\") "
Oct 30 22:55:03.862967 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.862944    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume" pod "5e254661-d49d-4325-96f5-cde04db4358b" (UID: "5e254661-d49d-4325-96f5-cde04db4358b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:03.864618 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.864594    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w" pod "5e254661-d49d-4325-96f5-cde04db4358b" (UID: "5e254661-d49d-4325-96f5-cde04db4358b"). InnerVolumeSpecName "kube-api-access-jdt8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:03.964071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.964039    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdt8w\" (UniqueName: \"kubernetes.io/projected/5e254661-d49d-4325-96f5-cde04db4358b-kube-api-access-jdt8w\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:03.964071 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:03.964066    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e254661-d49d-4325-96f5-cde04db4358b-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:04.608461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.608420    4882 generic.go:356] "Generic (PLEG): container finished" podID="5e254661-d49d-4325-96f5-cde04db4358b" containerID="44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37" exitCode=0
Oct 30 22:55:04.608692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.608472    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:55:04.608692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.608505    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" event={"ID":"5e254661-d49d-4325-96f5-cde04db4358b","Type":"ContainerDied","Data":"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"}
Oct 30 22:55:04.608692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.608564    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" event={"ID":"5e254661-d49d-4325-96f5-cde04db4358b","Type":"ContainerDied","Data":"4fe42fae172f6a526d80c700420a719d163295858a5b5a141544ae13d868ee82"}
Oct 30 22:55:04.608692 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.608608    4882 scope.go:122] "RemoveContainer" containerID="44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"
Oct 30 22:55:04.618630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.618576    4882 scope.go:122] "RemoveContainer" containerID="44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"
Oct 30 22:55:04.618926 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:55:04.618905    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37\": container with ID starting with 44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37 not found: ID does not exist" containerID="44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"
Oct 30 22:55:04.618998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.618935    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37"} err="failed to get container status \"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37\": rpc error: code = NotFound desc = could not find container \"44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37\": container with ID starting with 44072aad90afe845fdf99d59be5783efb3f7db28b5fe4ce2822d3f4929c01a37 not found: ID does not exist"
Oct 30 22:55:04.774115 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.774061    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.132.1.97:50052\" within 1s: context deadline exceeded"
Oct 30 22:55:04.774722 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.774164    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"
Oct 30 22:55:04.774722 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:55:04.774236    4882 logging.go:55] [core] [Channel #9532 SubChannel #9533]grpc: addrConn.createTransport failed to connect to {Addr: "10.132.1.97:50052", ServerName: "10.132.1.97:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.132.1.97:50052: operation was canceled"
Oct 30 22:55:04.775836 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.775814    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:55:04.778312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.778285    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-extmod-b-9af9e7-5f578cc4c-9xd7z"]
Oct 30 22:55:04.942306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:04.942226    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e254661-d49d-4325-96f5-cde04db4358b" path="/var/lib/kubelet/pods/5e254661-d49d-4325-96f5-cde04db4358b/volumes"
Oct 30 22:55:08.686794 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:08.686747    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing" containerID="cri-o://069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e" gracePeriod=25
Oct 30 22:55:08.964846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:08.964757    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:14.050899 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:14.050863    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"]
Oct 30 22:55:14.101252 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:14.101217    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8fa93289-sgjvf"]
Oct 30 22:55:14.942042 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:14.942002    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52296f8f-9157-49df-aa82-b56f2f04e8a1" path="/var/lib/kubelet/pods/52296f8f-9157-49df-aa82-b56f2f04e8a1/volumes"
Oct 30 22:55:15.058134 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:15.050266    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"]
Oct 30 22:55:15.099913 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:15.099871    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-10fef9d9-pnxfq"]
Oct 30 22:55:16.941380 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:16.941344    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed491cc6-f903-439f-80a0-4bf0e3e38ca1" path="/var/lib/kubelet/pods/ed491cc6-f903-439f-80a0-4bf0e3e38ca1/volumes"
Oct 30 22:55:18.965074 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:18.965037    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:20.806012 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:55:20.805976    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:55:22.116351 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:22.116320    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"]
Oct 30 22:55:22.165821 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:22.165778    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-6fc60c-fl7n7"]
Oct 30 22:55:22.942144 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:22.942108    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" path="/var/lib/kubelet/pods/1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f/volumes"
Oct 30 22:55:25.834718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:25.834684    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:55:28.942814 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:28.942747    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:55:29.115901 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.115864    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\" (UniqueName: \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\") pod \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") "
Oct 30 22:55:29.116082 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.115970    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\") pod \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\" (UID: \"4d9d3c53-392f-47b5-a0ea-ba090268ed24\") "
Oct 30 22:55:29.116379 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.116350    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume" pod "4d9d3c53-392f-47b5-a0ea-ba090268ed24" (UID: "4d9d3c53-392f-47b5-a0ea-ba090268ed24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:55:29.118138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.118114    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788" pod "4d9d3c53-392f-47b5-a0ea-ba090268ed24" (UID: "4d9d3c53-392f-47b5-a0ea-ba090268ed24"). InnerVolumeSpecName "kube-api-access-42788". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:55:29.218029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.217941    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-42788\" (UniqueName: \"kubernetes.io/projected/4d9d3c53-392f-47b5-a0ea-ba090268ed24-kube-api-access-42788\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.218029 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.217979    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d9d3c53-392f-47b5-a0ea-ba090268ed24-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:55:29.744646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.744612    4882 generic.go:356] "Generic (PLEG): container finished" podID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerID="069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e" exitCode=0
Oct 30 22:55:29.744862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.744656    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"
Oct 30 22:55:29.744862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.744659    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" event={"ID":"4d9d3c53-392f-47b5-a0ea-ba090268ed24","Type":"ContainerDied","Data":"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"}
Oct 30 22:55:29.744862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.744797    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp" event={"ID":"4d9d3c53-392f-47b5-a0ea-ba090268ed24","Type":"ContainerDied","Data":"8ffb837dac1d73a7d14c53486c536f1697c571ea9f67249c4bc3c95546dab250"}
Oct 30 22:55:29.744862 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.744832    4882 scope.go:122] "RemoveContainer" containerID="069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"
Oct 30 22:55:29.754797 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.754778    4882 scope.go:122] "RemoveContainer" containerID="069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"
Oct 30 22:55:29.755096 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:55:29.755077    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e\": container with ID starting with 069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e not found: ID does not exist" containerID="069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"
Oct 30 22:55:29.755154 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.755107    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e"} err="failed to get container status \"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e\": rpc error: code = NotFound desc = could not find container \"069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e\": container with ID starting with 069f964f9a076239f6051520f0f4d5ecdabb7ab6d3cdb3eebdc6d85d13d0c96e not found: ID does not exist"
Oct 30 22:55:29.918707 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.918664    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:55:29.971422 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:29.971378    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-6fc60c-595449d895-8cvvp"]
Oct 30 22:55:30.942446 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:30.942406    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" path="/var/lib/kubelet/pods/4d9d3c53-392f-47b5-a0ea-ba090268ed24/volumes"
Oct 30 22:55:40.902976 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:40.902936    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:55:40.955396 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:40.955360    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:55:45.903672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:45.903579    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing" containerID="cri-o://65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0" gracePeriod=25
Oct 30 22:55:45.956294 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:45.956243    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing" containerID="cri-o://445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550" gracePeriod=25
Oct 30 22:55:46.110387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:46.110344    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:55:50.809061 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:55:50.809019    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:55:53.166997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:53.166960    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:53.167426 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:53.166960    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:55:58.812930 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:58.812897    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 22:55:58.821535 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:55:58.821503    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 22:56:00.857760 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:00.857722    4882 scope.go:122] "RemoveContainer" containerID="7ed5ef07bd58fd63b37c8fbebab9732c684970cb46c1367166a75c2dfe8750eb"
Oct 30 22:56:00.868452 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:00.868427    4882 scope.go:122] "RemoveContainer" containerID="e67a81f0ad030700ec5c89ac4d9357e946232703d1a5a23a0b1328da8146743b"
Oct 30 22:56:00.877947 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:00.877925    4882 scope.go:122] "RemoveContainer" containerID="2e826b4f5dccd5f737907eae107d9fcdd5e0165a4ea3761638bdbdaec3e53ba0"
Oct 30 22:56:03.167003 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:03.166963    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:03.167432 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:03.166969    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Oct 30 22:56:06.088047 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088006    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"]
Oct 30 22:56:06.088567 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088549    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing"
Oct 30 22:56:06.088641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088569    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing"
Oct 30 22:56:06.088641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088607    4882 cpu_manager.go:414] "containerMap: removing container" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:56:06.088641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088617    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:56:06.088641 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088641    4882 cpu_manager.go:414] "containerMap: removing container" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088648    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088663    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" containerName="revoke-keys"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088668    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" containerName="revoke-keys"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088738    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e254661-d49d-4325-96f5-cde04db4358b" containerName="payload-pre-processing"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088749    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ca8f7b5-09cd-4b38-8548-c0738a5b3d1f" containerName="revoke-keys"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088758    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f04c653-84be-4ae7-94c7-99a00323d294" containerName="istio-proxy"
Oct 30 22:56:06.088786 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.088766    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="4d9d3c53-392f-47b5-a0ea-ba090268ed24" containerName="payload-pre-processing"
Oct 30 22:56:06.092228 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.092194    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.150001 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.149960    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"]
Oct 30 22:56:06.171331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.170414    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gj49s\" (UniqueName: \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.171331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.170524    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.171331 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.170576    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.199440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.199395    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Oct 30 22:56:06.199684 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.199597    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-mlvtf\""
Oct 30 22:56:06.220545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.220512    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:56:06.253468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.253441    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:56:06.271453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.271410    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gj49s\" (UniqueName: \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.271672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.271486    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.271672 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.271531    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.272201 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.272173    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.273903 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.273883    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.335345 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.335305    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gj49s\" (UniqueName: \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\") pod \"maas-api-revoke-sub-03771bea-tqfmh\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.372431 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372395    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\") pod \"f0958b46-bb85-422d-8b90-8212df761c7d\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") "
Oct 30 22:56:06.372663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372449    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\") pod \"f0810636-154a-461e-a74f-46e0e456b172\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") "
Oct 30 22:56:06.372663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372515    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\" (UniqueName: \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\") pod \"f0958b46-bb85-422d-8b90-8212df761c7d\" (UID: \"f0958b46-bb85-422d-8b90-8212df761c7d\") "
Oct 30 22:56:06.372663 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372577    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\" (UniqueName: \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\") pod \"f0810636-154a-461e-a74f-46e0e456b172\" (UID: \"f0810636-154a-461e-a74f-46e0e456b172\") "
Oct 30 22:56:06.372882 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372849    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume" pod "f0958b46-bb85-422d-8b90-8212df761c7d" (UID: "f0958b46-bb85-422d-8b90-8212df761c7d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:06.373040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.372930    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume" pod "f0810636-154a-461e-a74f-46e0e456b172" (UID: "f0810636-154a-461e-a74f-46e0e456b172"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:06.373040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.373016    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0958b46-bb85-422d-8b90-8212df761c7d-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:06.373040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.373031    4882 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f0810636-154a-461e-a74f-46e0e456b172-plugins-config-volume\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:06.374816 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.374792    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w" pod "f0958b46-bb85-422d-8b90-8212df761c7d" (UID: "f0958b46-bb85-422d-8b90-8212df761c7d"). InnerVolumeSpecName "kube-api-access-bzc6w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:06.374944 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.374833    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6" pod "f0810636-154a-461e-a74f-46e0e456b172" (UID: "f0810636-154a-461e-a74f-46e0e456b172"). InnerVolumeSpecName "kube-api-access-ccdw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:06.411981 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.411945    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:06.474062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.474027    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bzc6w\" (UniqueName: \"kubernetes.io/projected/f0958b46-bb85-422d-8b90-8212df761c7d-kube-api-access-bzc6w\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:06.474062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.474064    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccdw6\" (UniqueName: \"kubernetes.io/projected/f0810636-154a-461e-a74f-46e0e456b172-kube-api-access-ccdw6\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:06.786430 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:56:06.786398    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bac3add_032d_43ff_b41c_cb0a1a6f418e.slice/crio-56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7 WatchSource:0}: Error finding container 56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7: Status 404 returned error can't find the container with id 56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7
Oct 30 22:56:06.788483 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.788453    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"]
Oct 30 22:56:06.960998 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.960963    4882 generic.go:356] "Generic (PLEG): container finished" podID="f0810636-154a-461e-a74f-46e0e456b172" containerID="445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550" exitCode=0
Oct 30 22:56:06.961185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.961012    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"
Oct 30 22:56:06.961185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.961053    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" event={"ID":"f0810636-154a-461e-a74f-46e0e456b172","Type":"ContainerDied","Data":"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"}
Oct 30 22:56:06.961185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.961108    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m" event={"ID":"f0810636-154a-461e-a74f-46e0e456b172","Type":"ContainerDied","Data":"91d47c94b4043ded6942f765c13546b54e3f192d3078144cb75a0b09c93969f3"}
Oct 30 22:56:06.961185 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.961134    4882 scope.go:122] "RemoveContainer" containerID="445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"
Oct 30 22:56:06.962545 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.962515    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh" event={"ID":"6bac3add-032d-43ff-b41c-cb0a1a6f418e","Type":"ContainerStarted","Data":"56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7"}
Oct 30 22:56:06.964109 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.964081    4882 generic.go:356] "Generic (PLEG): container finished" podID="f0958b46-bb85-422d-8b90-8212df761c7d" containerID="65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0" exitCode=0
Oct 30 22:56:06.964225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.964155    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" event={"ID":"f0958b46-bb85-422d-8b90-8212df761c7d","Type":"ContainerDied","Data":"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"}
Oct 30 22:56:06.964225 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.964212    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw" event={"ID":"f0958b46-bb85-422d-8b90-8212df761c7d","Type":"ContainerDied","Data":"467e63b6ed999eb657c79173ef7f8ab01e4d38754e2bbbd44fe8d52850969dc7"}
Oct 30 22:56:06.964312 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.964169    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"
Oct 30 22:56:06.971341 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.971320    4882 scope.go:122] "RemoveContainer" containerID="445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"
Oct 30 22:56:06.971664 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:56:06.971639    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550\": container with ID starting with 445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550 not found: ID does not exist" containerID="445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"
Oct 30 22:56:06.971777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.971671    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550"} err="failed to get container status \"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550\": rpc error: code = NotFound desc = could not find container \"445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550\": container with ID starting with 445faf3dc572788b7e95eb07e347a380f17bf025a373229cee82c1986a5d8550 not found: ID does not exist"
Oct 30 22:56:06.971777 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.971691    4882 scope.go:122] "RemoveContainer" containerID="65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"
Oct 30 22:56:06.980970 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.980948    4882 scope.go:122] "RemoveContainer" containerID="65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"
Oct 30 22:56:06.981250 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:56:06.981229    4882 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0\": container with ID starting with 65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0 not found: ID does not exist" containerID="65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"
Oct 30 22:56:06.981326 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:06.981257    4882 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0"} err="failed to get container status \"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0\": rpc error: code = NotFound desc = could not find container \"65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0\": container with ID starting with 65c17a62825a650cf8cdde8542b1e780b102b7646b99827f3a2547fc94a083b0 not found: ID does not exist"
Oct 30 22:56:07.130007 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.129960    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:56:07.136034 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.136005    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w0-c29159-78bc4bbfb7-7dh9m"]
Oct 30 22:56:07.247805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.247765    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:56:07.254360 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.254331    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c29159-c798bb7d9-w4nlw"]
Oct 30 22:56:07.972215 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.972125    4882 generic.go:356] "Generic (PLEG): container finished" podID="6bac3add-032d-43ff-b41c-cb0a1a6f418e" containerID="89f3e2ac69634a9f525e2cdce75a5a23d7f2b7cb941681005d3297b7c4c7d3c3" exitCode=0
Oct 30 22:56:07.972387 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:07.972221    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh" event={"ID":"6bac3add-032d-43ff-b41c-cb0a1a6f418e","Type":"ContainerDied","Data":"89f3e2ac69634a9f525e2cdce75a5a23d7f2b7cb941681005d3297b7c4c7d3c3"}
Oct 30 22:56:08.941696 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:08.941663    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0810636-154a-461e-a74f-46e0e456b172" path="/var/lib/kubelet/pods/f0810636-154a-461e-a74f-46e0e456b172/volumes"
Oct 30 22:56:08.942105 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:08.942083    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" path="/var/lib/kubelet/pods/f0958b46-bb85-422d-8b90-8212df761c7d/volumes"
Oct 30 22:56:09.109856 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.109829    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:09.200287 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.200188    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\") pod \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") "
Oct 30 22:56:09.200488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.200298    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\" (UniqueName: \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\") pod \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") "
Oct 30 22:56:09.200488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.200330    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\") pod \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\" (UID: \"6bac3add-032d-43ff-b41c-cb0a1a6f418e\") "
Oct 30 22:56:09.200631 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.200573    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca" pod "6bac3add-032d-43ff-b41c-cb0a1a6f418e" (UID: "6bac3add-032d-43ff-b41c-cb0a1a6f418e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Oct 30 22:56:09.200729 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.200709    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-openshift-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:09.202558 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.202525    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s" pod "6bac3add-032d-43ff-b41c-cb0a1a6f418e" (UID: "6bac3add-032d-43ff-b41c-cb0a1a6f418e"). InnerVolumeSpecName "kube-api-access-gj49s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:09.202757 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.202579    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca" pod "6bac3add-032d-43ff-b41c-cb0a1a6f418e" (UID: "6bac3add-032d-43ff-b41c-cb0a1a6f418e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:09.301295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.301258    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gj49s\" (UniqueName: \"kubernetes.io/projected/6bac3add-032d-43ff-b41c-cb0a1a6f418e-kube-api-access-gj49s\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:09.301295 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.301291    4882 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bac3add-032d-43ff-b41c-cb0a1a6f418e-maas-api-xks-ca\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:09.985811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.985773    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh"
Oct 30 22:56:09.985811 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.985780    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-03771bea-tqfmh" event={"ID":"6bac3add-032d-43ff-b41c-cb0a1a6f418e","Type":"ContainerDied","Data":"56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7"}
Oct 30 22:56:09.986270 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:09.985835    4882 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="56a5dec1955b33771c23b9487935b3bc3d15cb13e6b6f633fee8388c11c18ef7"
Oct 30 22:56:13.833638 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:13.833596    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" containerID="cri-o://1dba39d3023d4c7d15661ec3251d091703d1e8cb1de86e56069428d90feb3bcb" gracePeriod=45
Oct 30 22:56:13.841217 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:13.841183    4882 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main" containerID="cri-o://c0a9df4741d406b358c4bf62347782bc59fc7624d3dfb69bac47093257ede46d" gracePeriod=45
Oct 30 22:56:13.909016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:13.908979    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main" probeResult="failure" output="Get \"https://10.132.0.46:8000/ready\": dial tcp 10.132.0.46:8000: connect: connection refused"
Oct 30 22:56:13.910390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:13.910365    4882 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main" probeResult="failure" output="Get \"https://10.132.0.48:8000/ready\": dial tcp 10.132.0.48:8000: connect: connection refused"
Oct 30 22:56:14.019978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.019945    4882 generic.go:356] "Generic (PLEG): container finished" podID="1bdee079-40db-4af7-8b85-0bde190d3523" containerID="c0a9df4741d406b358c4bf62347782bc59fc7624d3dfb69bac47093257ede46d" exitCode=0
Oct 30 22:56:14.020168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.020014    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" event={"ID":"1bdee079-40db-4af7-8b85-0bde190d3523","Type":"ContainerDied","Data":"c0a9df4741d406b358c4bf62347782bc59fc7624d3dfb69bac47093257ede46d"}
Oct 30 22:56:14.022231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.022124    4882 generic.go:356] "Generic (PLEG): container finished" podID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerID="1dba39d3023d4c7d15661ec3251d091703d1e8cb1de86e56069428d90feb3bcb" exitCode=0
Oct 30 22:56:14.022231 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.022186    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" event={"ID":"c62c3cf7-5a2c-4c05-987c-4c2cf8188928","Type":"ContainerDied","Data":"1dba39d3023d4c7d15661ec3251d091703d1e8cb1de86e56069428d90feb3bcb"}
Oct 30 22:56:14.092796 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.092728    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 22:56:14.119990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.119960    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 22:56:14.249461 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249429    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.249678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249479    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.249678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249505    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.249678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249520    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.249678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249551    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.249678 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249577    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.249997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249743    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.249997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249779    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.249997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249821    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\" (UniqueName: \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.249997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249888    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.250222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250090    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.250222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250097    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.250222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250106    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.250439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250403    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.250439 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.249861    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\") pod \"1bdee079-40db-4af7-8b85-0bde190d3523\" (UID: \"1bdee079-40db-4af7-8b85-0bde190d3523\") "
Oct 30 22:56:14.250563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250473    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\" (UniqueName: \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.250563 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250536    4882 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\") pod \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\" (UID: \"c62c3cf7-5a2c-4c05-987c-4c2cf8188928\") "
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250907    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250931    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250946    4882 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-home\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250958    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250971    4882 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-tmp-dir\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.251033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.250998    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.252289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.252172    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.252289 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.252219    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Oct 30 22:56:14.252469 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.252452    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:14.252785 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.252761    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Oct 30 22:56:14.253107 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.253086    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm" pod "c62c3cf7-5a2c-4c05-987c-4c2cf8188928" (UID: "c62c3cf7-5a2c-4c05-987c-4c2cf8188928"). InnerVolumeSpecName "kube-api-access-nqvtm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:14.253419 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.253404    4882 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv" pod "1bdee079-40db-4af7-8b85-0bde190d3523" (UID: "1bdee079-40db-4af7-8b85-0bde190d3523"). InnerVolumeSpecName "kube-api-access-wjmdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Oct 30 22:56:14.352440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352359    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1bdee079-40db-4af7-8b85-0bde190d3523-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352401    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjmdv\" (UniqueName: \"kubernetes.io/projected/1bdee079-40db-4af7-8b85-0bde190d3523-kube-api-access-wjmdv\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352418    4882 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqvtm\" (UniqueName: \"kubernetes.io/projected/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-kube-api-access-nqvtm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352440 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352433    4882 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-model-cache\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352448    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1bdee079-40db-4af7-8b85-0bde190d3523-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352461    4882 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-dshm\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:14.352718 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:14.352474    4882 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c62c3cf7-5a2c-4c05-987c-4c2cf8188928-tls-certs\") on node \"ip-10-0-141-82.us-west-2.compute.internal\" DevicePath \"\""
Oct 30 22:56:15.028942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.028907    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"
Oct 30 22:56:15.028942 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.028927    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k" event={"ID":"1bdee079-40db-4af7-8b85-0bde190d3523","Type":"ContainerDied","Data":"33057e4d812b0b85aca3d7cc1285c034568fb77b0dadceb16884066adce180d2"}
Oct 30 22:56:15.029487 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.028986    4882 scope.go:122] "RemoveContainer" containerID="c0a9df4741d406b358c4bf62347782bc59fc7624d3dfb69bac47093257ede46d"
Oct 30 22:56:15.030693 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.030670    4882 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"
Oct 30 22:56:15.030813 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.030666    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt" event={"ID":"c62c3cf7-5a2c-4c05-987c-4c2cf8188928","Type":"ContainerDied","Data":"d46f27ae1f8136a018973e1e9f419240f39e60c37781b6387ab34a2d3f802f42"}
Oct 30 22:56:15.038745 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.038725    4882 scope.go:122] "RemoveContainer" containerID="1dba39d3023d4c7d15661ec3251d091703d1e8cb1de86e56069428d90feb3bcb"
Oct 30 22:56:15.199222 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.199114    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 22:56:15.248159 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.248119    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/simulator-w0-9a5623-kserve-856b659b97-k6s7k"]
Oct 30 22:56:15.317680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.317641    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 22:56:15.321027 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:15.321001    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-9a5623/premium-w0-9a5623-kserve-78b4b7457b-7rrqt"]
Oct 30 22:56:16.941174 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:16.941131    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" path="/var/lib/kubelet/pods/1bdee079-40db-4af7-8b85-0bde190d3523/volumes"
Oct 30 22:56:16.941859 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:16.941765    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" path="/var/lib/kubelet/pods/c62c3cf7-5a2c-4c05-987c-4c2cf8188928/volumes"
Oct 30 22:56:20.806142 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:56:20.806101    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:56:23.027711 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:23.027675    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:56:31.253971 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:31.253941    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xfkcg_8c8af1fe-acc2-41f4-9146-c47b97930a8a/kuadrant-console-plugin/0.log"
Oct 30 22:56:32.543843 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:32.543814    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2slh_eb806f00-3ab1-40e7-b4a2-7f7f131b7338/limitador/0.log"
Oct 30 22:56:34.175772 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:34.175742    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:56:34.666679 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:34.666643    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:56:35.356943 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:35.356914    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:56:35.597251 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:35.597220    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:56:35.862485 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:35.862445    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:56:36.692784 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:36.692746    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-7c9b87d574-w2slh"]
Oct 30 22:56:47.770979 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:47.770944    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:56:48.395268 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:48.395234    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:56:49.095210 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:49.095174    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:56:49.327474 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:49.327436    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:56:49.594453 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:49.594420    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:56:50.806733 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:56:50.806699    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:56:50.847846 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:50.847815    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-03771bea-tqfmh_6bac3add-032d-43ff-b41c-cb0a1a6f418e/revoke-keys/0.log"
Oct 30 22:56:51.291581 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:51.291550    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-462f57d1-ptfkv_615a4250-0077-47bf-a165-1f30f28c34a8/revoke-keys/0.log"
Oct 30 22:56:51.512808 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:51.512775    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-82285251-k586m_3da63bcb-26b5-4a4f-820d-b85a44f2f3d0/revoke-keys/0.log"
Oct 30 22:56:51.736997 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:51.736963    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-8977f241-4kpvm_be6ab3ad-6064-4bfd-9613-9f004c771730/revoke-keys/0.log"
Oct 30 22:56:51.968747 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:51.968719    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-a1a785c4-x66bt_450d4462-660e-4ff3-849f-817528a4ace5/revoke-keys/0.log"
Oct 30 22:56:52.196039 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:52.196008    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-d8ab1463-xffmg_77cb3f32-d9d4-4907-8795-9b6a83114aee/revoke-keys/0.log"
Oct 30 22:56:52.422805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:52.422770    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-kzzxd_3e7d26f3-57e7-4aa4-9ac2-75ac97d1ecf2/postgres/0.log"
Oct 30 22:56:54.870406 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:54.870378    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xfkcg_8c8af1fe-acc2-41f4-9146-c47b97930a8a/kuadrant-console-plugin/0.log"
Oct 30 22:56:56.171611 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:56.171559    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2slh_eb806f00-3ab1-40e7-b4a2-7f7f131b7338/limitador/0.log"
Oct 30 22:56:57.959157 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:56:57.959128    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-pbxxx_527a8dd9-c14b-45b0-b54e-a632c59f9c1e/kube-auth-proxy/0.log"
Oct 30 22:57:00.925698 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:00.925666    4882 scope.go:122] "RemoveContainer" containerID="796dff028a5f6b5fcd4d908492ead515be86665d080a9b431eeb7e85d737fea0"
Oct 30 22:57:00.935254 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:00.935230    4882 scope.go:122] "RemoveContainer" containerID="43ba58a5c99abf402874aea3c0e728f51ea64b4dfce436c3e70a734d696767aa"
Oct 30 22:57:00.944543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:00.944517    4882 scope.go:122] "RemoveContainer" containerID="03f1e00454a818bd68f7be606e97ae7b6ae23624be8accb6573b7f2f1160e784"
Oct 30 22:57:02.068290 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.068248    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"]
Oct 30 22:57:02.118408 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.118378    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"]
Oct 30 22:57:02.120400 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.120388    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-82285251-k586m"]
Oct 30 22:57:02.122096 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.122072    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1a785c4-x66bt"]
Oct 30 22:57:02.941568 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.941533    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3da63bcb-26b5-4a4f-820d-b85a44f2f3d0" path="/var/lib/kubelet/pods/3da63bcb-26b5-4a4f-820d-b85a44f2f3d0/volumes"
Oct 30 22:57:02.942033 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:02.942017    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="450d4462-660e-4ff3-849f-817528a4ace5" path="/var/lib/kubelet/pods/450d4462-660e-4ff3-849f-817528a4ace5/volumes"
Oct 30 22:57:20.806770 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:57:20.806726    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:57:21.724912 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:21.724877    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-6pwrc_d97ac33e-24e8-4256-a3cd-154231e10553/global-pull-secret-syncer/0.log"
Oct 30 22:57:25.826168 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:25.826132    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-sxl4l_2bf2774c-7d35-4bff-98ae-489cc598b066/konnectivity-agent/0.log"
Oct 30 22:57:26.162710 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:26.162653    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:26.361141 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:26.361108    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:26.971989 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:26.971950    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-82.us-west-2.compute.internal_845d50ae5087e4cd62e2c4035d1ecf3f/haproxy/0.log"
Oct 30 22:57:27.281624 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:27.281494    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:27.481181 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:27.481139    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:27.736462 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:27.736433    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:28.046704 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:28.046612    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:30.244980 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:30.244935    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:30.371622 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:30.371572    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:30.557564 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:30.557467    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:30.675520 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:30.675487    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:32.743390 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.743350    4882 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"]
Oct 30 22:57:32.744008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.743988    4882 cpu_manager.go:414] "containerMap: removing container" podUID="6bac3add-032d-43ff-b41c-cb0a1a6f418e" containerName="revoke-keys"
Oct 30 22:57:32.744062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744011    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bac3add-032d-43ff-b41c-cb0a1a6f418e" containerName="revoke-keys"
Oct 30 22:57:32.744062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744041    4882 cpu_manager.go:414] "containerMap: removing container" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main"
Oct 30 22:57:32.744062 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744048    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744064    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744072    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744101    4882 cpu_manager.go:414] "containerMap: removing container" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744108    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744123    4882 cpu_manager.go:414] "containerMap: removing container" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing"
Oct 30 22:57:32.744164 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744131    4882 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing"
Oct 30 22:57:32.744362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744219    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="1bdee079-40db-4af7-8b85-0bde190d3523" containerName="main"
Oct 30 22:57:32.744362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744235    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0810636-154a-461e-a74f-46e0e456b172" containerName="payload-pre-processing"
Oct 30 22:57:32.744362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744247    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bac3add-032d-43ff-b41c-cb0a1a6f418e" containerName="revoke-keys"
Oct 30 22:57:32.744362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744258    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="c62c3cf7-5a2c-4c05-987c-4c2cf8188928" containerName="main"
Oct 30 22:57:32.744362 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.744270    4882 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0958b46-bb85-422d-8b90-8212df761c7d" containerName="payload-processing"
Oct 30 22:57:32.747435 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.747413    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.800437 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.800401    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"]
Oct 30 22:57:32.848498 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.848470    4882 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-v9zbs\"/\"default-dockercfg-r6fhw\""
Oct 30 22:57:32.876507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.876467    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-podres\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.876738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.876523    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-lib-modules\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.876738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.876621    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-sys\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.876738 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.876645    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-proc\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.876911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.876732    4882 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rcsq\" (UniqueName: \"kubernetes.io/projected/ef5b3343-c52a-4230-9d1e-e328797be176-kube-api-access-8rcsq\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.898758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.898720    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v9zbs\"/\"openshift-service-ca.crt\""
Oct 30 22:57:32.898758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.898743    4882 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-v9zbs\"/\"kube-root-ca.crt\""
Oct 30 22:57:32.977278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977246    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-lib-modules\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977284    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-sys\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977302    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-proc\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977331    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rcsq\" (UniqueName: \"kubernetes.io/projected/ef5b3343-c52a-4230-9d1e-e328797be176-kube-api-access-8rcsq\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977395    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-lib-modules\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977399    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-sys\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977405    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-proc\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977488 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977454    4882 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-podres\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:32.977895 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:32.977539    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ef5b3343-c52a-4230-9d1e-e328797be176-podres\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:33.039825 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.039745    4882 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rcsq\" (UniqueName: \"kubernetes.io/projected/ef5b3343-c52a-4230-9d1e-e328797be176-kube-api-access-8rcsq\") pod \"perf-node-gather-daemonset-t8p8w\" (UID: \"ef5b3343-c52a-4230-9d1e-e328797be176\") " pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:33.060689 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.060657    4882 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:33.251124 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.251094    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xfkcg_8c8af1fe-acc2-41f4-9146-c47b97930a8a/kuadrant-console-plugin/0.log"
Oct 30 22:57:33.431087 ip-10-0-141-82 kubenswrapper[4882]: W1030 22:57:33.431045    4882 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podef5b3343_c52a_4230_9d1e_e328797be176.slice/crio-a20cf9c3d64cb8c4fe51396d405882d183522d585a733e9144051a0a947a48e8 WatchSource:0}: Error finding container a20cf9c3d64cb8c4fe51396d405882d183522d585a733e9144051a0a947a48e8: Status 404 returned error can't find the container with id a20cf9c3d64cb8c4fe51396d405882d183522d585a733e9144051a0a947a48e8
Oct 30 22:57:33.433883 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.433856    4882 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"]
Oct 30 22:57:33.478008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.477973    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w" event={"ID":"ef5b3343-c52a-4230-9d1e-e328797be176","Type":"ContainerStarted","Data":"a20cf9c3d64cb8c4fe51396d405882d183522d585a733e9144051a0a947a48e8"}
Oct 30 22:57:33.522508 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.522475    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:33.859255 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:33.859228    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:34.301510 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.301423    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:34.488263 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.488220    4882 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w" event={"ID":"ef5b3343-c52a-4230-9d1e-e328797be176","Type":"ContainerStarted","Data":"ea0010314209083408750ba89269580a783b551f27401b8a083c6310a1e87e49"}
Oct 30 22:57:34.488507 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.488411    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:34.611621 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.611540    4882 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w" podStartSLOduration=2.6115233829999998 podStartE2EDuration="2.611523383s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 22:57:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 22:57:34.608501414 +0000 UTC m=+4777.781825841" watchObservedRunningTime="2026-10-30 22:57:34.611523383 +0000 UTC m=+4777.784847811"
Oct 30 22:57:34.748977 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.748939    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:34.793826 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.793800    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-7c9b87d574-w2slh_eb806f00-3ab1-40e7-b4a2-7f7f131b7338/limitador/0.log"
Oct 30 22:57:34.843990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:34.843965    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:35.183297 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:35.183261    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:37.301212 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:37.301048    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:37.462595 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:37.462551    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:37.650428 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:37.650397    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:37.844990 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:37.844957    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:37.907403 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:37.907313    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/alertmanager/0.log"
Oct 30 22:57:38.316978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:38.316900    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/config-reloader/0.log"
Oct 30 22:57:38.724805 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:38.724767    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/kube-rbac-proxy-web/0.log"
Oct 30 22:57:39.133468 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:39.133437    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/kube-rbac-proxy/0.log"
Oct 30 22:57:39.549272 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:39.549188    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/kube-rbac-proxy-metric/0.log"
Oct 30 22:57:39.951306 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:39.951273    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/prom-label-proxy/0.log"
Oct 30 22:57:40.359855 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:40.359820    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1de6cc72-b70a-438b-940c-f4581274b86b/init-config-reloader/0.log"
Oct 30 22:57:40.508911 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:40.508881    4882 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-v9zbs/perf-node-gather-daemonset-t8p8w"
Oct 30 22:57:40.640651 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:40.640568    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:40.946375 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:40.946296    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-bzpq4_3fc14e3f-e24b-4525-b02f-f762c0f2afc8/manager/0.log"
Oct 30 22:57:41.050490 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.050457    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"]
Oct 30 22:57:41.053829 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.053804    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8977f241-4kpvm"]
Oct 30 22:57:41.182614 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.182560    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/0.log"
Oct 30 22:57:41.198016 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.197919    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-state-metrics/1.log"
Oct 30 22:57:41.506076 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.505980    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-rbac-proxy-main/0.log"
Oct 30 22:57:41.706002 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.705963    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:41.915494 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.915457    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-7pfdn_773ef005-be9c-4f87-b282-c5fad993d143/kube-rbac-proxy-self/0.log"
Oct 30 22:57:41.971748 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:41.971717    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-6466c554d8-sdnsl_f7d5fad6-516c-4281-98f5-b8df1dc8c40e/manager/0.log"
Oct 30 22:57:42.051041 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.051010    4882 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"]
Oct 30 22:57:42.101551 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.101517    4882 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-462f57d1-ptfkv"]
Oct 30 22:57:42.165207 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.165176    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:42.350845 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.350809    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-767dd57b75-mqc9s_33313b40-bb78-46ad-8229-08f46def4870/metrics-server/0.log"
Oct 30 22:57:42.458680 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.458640    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-bmxth_9f3c8bce-2e78-4a51-8636-8058e2857a96/server/0.log"
Oct 30 22:57:42.763995 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.763907    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-77f4f76594-fnwfz_7bb75240-abd5-43e1-aa11-7604abed835d/monitoring-plugin/0.log"
Oct 30 22:57:42.944040 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.944005    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="615a4250-0077-47bf-a165-1f30f28c34a8" path="/var/lib/kubelet/pods/615a4250-0077-47bf-a165-1f30f28c34a8/volumes"
Oct 30 22:57:42.944537 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:42.944501    4882 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be6ab3ad-6064-4bfd-9613-9f004c771730" path="/var/lib/kubelet/pods/be6ab3ad-6064-4bfd-9613-9f004c771730/volumes"
Oct 30 22:57:44.623211 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:44.623169    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:44.785646 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:44.785607    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:44.931008 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:44.930923    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-qx8nl_a946fc79-675f-41f6-b52f-882569f35dc2/manager/0.log"
Oct 30 22:57:45.092234 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:45.092205    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-929tg_c39beeb8-af5b-4396-be2b-80d1c65fb869/proxy/0.log"
Oct 30 22:57:45.649548 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:45.649511    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6b7ms_0ab31f1a-cea7-4422-83e6-d146cb9f920a/node-exporter/0.log"
Oct 30 22:57:46.116314 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:46.116284    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6b7ms_0ab31f1a-cea7-4422-83e6-d146cb9f920a/kube-rbac-proxy/0.log"
Oct 30 22:57:46.533250 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:46.533179    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-6b7ms_0ab31f1a-cea7-4422-83e6-d146cb9f920a/init-textfile/0.log"
Oct 30 22:57:46.944758 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:46.944723    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q4h9q_361370c9-679d-4243-9450-7a315ec4f459/kube-rbac-proxy-main/0.log"
Oct 30 22:57:47.362114 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:47.362073    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q4h9q_361370c9-679d-4243-9450-7a315ec4f459/kube-rbac-proxy-self/0.log"
Oct 30 22:57:47.771278 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:47.771186    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-q4h9q_361370c9-679d-4243-9450-7a315ec4f459/openshift-state-metrics/0.log"
Oct 30 22:57:48.195630 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:48.195600    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/prometheus/0.log"
Oct 30 22:57:48.604533 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:48.604501    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/config-reloader/0.log"
Oct 30 22:57:49.014539 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:49.014445    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/thanos-sidecar/0.log"
Oct 30 22:57:49.502030 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:49.502001    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/kube-rbac-proxy-web/0.log"
Oct 30 22:57:49.810532 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:49.810437    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-pbxxx_527a8dd9-c14b-45b0-b54e-a632c59f9c1e/kube-auth-proxy/0.log"
Oct 30 22:57:49.918343 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:49.918312    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/kube-rbac-proxy/0.log"
Oct 30 22:57:50.006337 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:50.006307    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-794f59cf45-pbxxx_527a8dd9-c14b-45b0-b54e-a632c59f9c1e/kube-auth-proxy/0.log"
Oct 30 22:57:50.325773 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:50.325744    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/kube-rbac-proxy-thanos/0.log"
Oct 30 22:57:50.733075 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:50.733043    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_697441e8-bc22-474a-9bc0-d8506d0b40da/init-config-reloader/0.log"
Oct 30 22:57:50.807534 ip-10-0-141-82 kubenswrapper[4882]: E1030 22:57:50.807488    4882 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-p2prz" containerName="extractor"
Oct 30 22:57:52.386994 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:52.386953    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fc9f56dfc-cxb5q_479b5a03-a117-4edf-8da2-92f43a614802/telemeter-client/0.log"
Oct 30 22:57:52.794557 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:52.794469    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fc9f56dfc-cxb5q_479b5a03-a117-4edf-8da2-92f43a614802/reload/0.log"
Oct 30 22:57:53.202031 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:53.201997    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6fc9f56dfc-cxb5q_479b5a03-a117-4edf-8da2-92f43a614802/kube-rbac-proxy/0.log"
Oct 30 22:57:53.618220 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:53.618191    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/thanos-query/0.log"
Oct 30 22:57:54.025213 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:54.025134    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/kube-rbac-proxy-web/0.log"
Oct 30 22:57:54.433543 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:54.433514    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/kube-rbac-proxy/0.log"
Oct 30 22:57:54.840842 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:54.840814    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/prom-label-proxy/0.log"
Oct 30 22:57:55.250138 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:55.250021    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/kube-rbac-proxy-rules/0.log"
Oct 30 22:57:55.663905 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:55.663881    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-fc58c9df8-bjk8x_2024b1bc-1853-477d-8ae2-8329bd3dbb8f/kube-rbac-proxy-metrics/0.log"
Oct 30 22:57:57.509978 ip-10-0-141-82 kubenswrapper[4882]: I1030 22:57:57.509941    4882 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-xfkcg_8c8af1fe-acc2-41f4-9146-c47b97930a8a/kuadrant-console-plugin/0.log"